Introduction

It is hard to imagine everyday life without access to a wide spectrum of various services and applications delivered via the Web. Web content is available across a wide range of platforms and networks, with responsive Web design and adaptive media becoming the de facto standard approaches to editing that content. Moreover, the proliferation of mobile devices such as smartphones and tablets, and emerging devices such as smart watches or smart wristbands, together with the advances in their capacity, functionality, and design, have changed the usage of these devices beyond making calls or sending and reading text messages, to usage scenarios such as reading e-mail, browsing news sites, social networks, online shopping, gaming, etc. For example, according to Deloitte, in 2016 31% of smartphone users did not make any traditional voice calls in a given week, which is in contrast to 25% in 2015 and only 4% in 2012, when the time spent was 12.13 min per day (The Huffington Post) [129, 130]. Such trends are further pushed by rapid developments in the field of wireless mobile communications, resulting in increased user requirements and expectations in terms of accessing a wide variety of Web services, that is, anywhere, anytime, and via multiple devices as discussed in Baraković and Skorin-Kapov [4, 5], Ickin et al. [47], or Stankiewicz and Jajszczyk [112].

According to Cisco [18], it is predicted that mobile data traffic will increase nearly tenfold between 2014 and 2019. In favour of this are the statistics by NetMarketShare [82] which show that in the last two years traffic generated by mobile Web browsing has increased by approximately 12.5%, and accounted for nearly 28.67% of all Web browsing (desktop and mobile) in August 2016. Also, Statcounter [131] stated that in October 2016 the percentage of pages loaded on mobile devices surpassed desktop and laptop computers for the first time. These trends are recognized by various stakeholders in the mobile Web application and service provisioning chain, such as network operators, device manufacturers, mobile Web designers, and all of those offering information, communication, business, or entertainment over the Web. Different application/content delivery options are available to end users, including native applications, mobile Web browsing content, or hybrid applications.

Given both the widespread use of Web browsing in various different contexts, and the continuous evolvement of content, applications, devices, networks, and usage scenarios, there is a clear need for ongoing research efforts in the domain of user research looking to understand what impacts the user experience and quality perception when interacting with various types of Web content. Numerous relevant studies have been previously published in this domain, and can for the most part be classified as belonging to one of two research communities. Within the field of human–computer interaction (HCI), user experience (UX) studies have addressed user preferences and experiences in relation to aspects such as the perceived usability and aesthetics of Web-related content. On the other hand, studies originating from the networking and telecommunications field are more recent, and have focused on analysing the impacts of network performance and waiting times (resulting from page/element loading times) on so-called Quality of Experience (QoE). Consequently, hitherto the fields of Web QoE and HCI-related Web studies have for the most part diverged (an exception are studies by Baraković [3], Baraković and Skorin-Kapov [5], Varela et al. [122]), although their aims are similar and directed towards a better understanding of user experience and overall satisfaction when accessing Web sites and applications.

QoE has been defined by Le Callet et al. [64] (in the scope of the EU COST action QualinetFootnote 1) as “the degree of delight or annoyance of the user of an application or service. It results from the fulfilment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the user’s personality and current state. In the context of communication services, QoE is influenced by service, content, device, application, and context of use”. Although the concept of QoE has been widely applied in the domain of multimedia communication systems, especially in terms of audio and video quality assessment, its application to the domain of Web browsing has received less attention, particularly when it comes to the mobile context. Web browsing is inherently interactive by nature, since users do not perceive it as a sequence of isolated page loadings, but rather as a sort of flow experience, that is, request-response session as given by Schatz et al. [106]. Accordingly, Hoßfeld et al. [44] have defined Web QoE as “the Quality of Experience of interactive services that are based on HTTP and accessed via a browser.” The International Telecommunications Union (ITU) has published a recommendation, given in [49], outlining subjective testing methods for assessing the users’ perceived quality for Web browsing in browser-based applications of different device classes. The recommendation defines a Web browsing session as being “an interactive information exchange between a user and one or more websites over a limited period of time, mediated via a web browsing application.”

A great deal of empirical research to date has focused on the domain of UX (for example, as discussed by Bargas-Avila and Hornbæk [7] or Hassenzahl and Monk [42, 43]), driven by a human-centred approach at the level of both theory and practice, as stated by Roto [95]. A thorough comparison of QoE and UX is given by Wechsung and De Moor [125], who highlight the theoretical-conceptual and methodological-practical differences. Although a number of similarities were identified, the differences between these two notions are profound and reflect in [125]:

  • origin—QoE comes from the telecommunications community while UX comes from the HCI community;

  • driving force—UX is human-centred, while QoE is considered to be primarily system- and technology-centred;

  • theoretical basis—UX has a strong influence from fields such as psychology, human factors, and HCI, that is, it has a strong theoretical basis, while QoE evolved in an application- and practice-driven way;

  • measurement and evaluation—UX draws qualitative research methods (and to a certain extent quantitative approaches), while QoE is based primarily on quantitative approaches; and

  • focus—UX is concentrated on experience, while QoE has typically focused on the quality formation process and features that contribute to the perception of quality.

In addition, the main focus of QoE is to evaluate quality perception and gather input to guide the optimization of technical parameters at different layers (application layer, network layer), while UX is focused on evaluating and understanding user experience and the process of experiencing. However, one of the most prominent differences among these two notions is that UX is aimed at understanding, while QoE is aimed at quantifying the relations and thus gaining understanding based on quantification. An important conclusion that Wechsung and De Moor draw in [125] is that bringing the concepts and methodologies from the UX field into the domain of QoE is needed to put the aforementioned holistic definition of QoE into practice. Recent advancement of the definition of QoE as discussed by Raake and Eggger [90] has pushed the theoretical foundations of QoE and reached towards the UX concept. We note the UX concept has been clarified in detail in a previously published White Paper [96].

Based on all previously mentioned, there is a clear necessity in modelling (quantification) and better understanding of the QoE when accessing Web sites and applications. The research community will consequently need to focus on exploring the factors that influence QoE, and understanding how they mutually correlate after quantifying and modelling their relations. As examples, knowledge in this domain can aid practitioners in terms of Web and product design (design of more appealing and usable Web sites), and network monitoring and management specialists (allocation of sufficient network resources so as to provide acceptable page loading times). Each of the different possible stakeholders can benefit from the structured survey given in this paper by focusing on the results drawn from papers targeting the corresponding domain of interest.

Given that extensive research has been conducted aimed at studying various aspects of Web UX and QoE, we identify the need for a structured overview of such studies. Our motivation lies in attempting to summarise and draw together key findings from existing research, as well as in providing valuable input for researchers in the field in terms of key challenges that remain to be addressed in future work on Web QoE. Our focus is thus on surveying QoE modelling approaches and identifying open issues and key challenges. Additionally, by briefly discussing relevant research addressed by the HCI community, the QoE researcher can benefit from insight into a broader scope, going beyond the more commonly addressed narrow scope of technical aspects and subjective perception of waiting times. The HCI/UX community has been working on issues of acceptance, experience, and crucial design factors extensively for a long time, resulting in a broad understanding. We believe that the literature discussed in this overview will benefit the QoE researcher looking to address multidimensional Web QoE. On the other hand, the HCI researcher can benefit from insights into research conducted in the QoE domain in terms of the impact of more technical factors on the user experience.

Methodology

The theoretical framework that we adopt in conducting this survey is as follows: factors influencing the QoE of Web browsing can be categorized as being either system-, context-, or human-related influence factors (IFs) (according to Le Callet et al. [64] and Reiter et al. [94]). Further, QoE is considered to be a multidimensional construct that can be decomposed into multiple perceptual QoE features, referring to “perceivable, recognizable, and nameable characteristics of the individual’s experience of a service which contributes to its quality” as given by Jekosch [132] and Le Callete et al. [64] (Fig. 1). We adopt this theoretical framework as a basis for systematically categorizing and analysing relevant literature (we note that Fig. 1 portrays a non-exhaustive list of example Web QoE IFs and features, extracted based on the surveyed literature). As will be discussed, a number these factors and features are also related to UX and have been well studied in the HCI domain, which is why we bring them into the discussion on QoE.

Fig. 1
figure 1

Theoretical framework as inspired by Le Callet et al. in [64]: QoE modelling scheme (examples of QoE influence factors (IFs) and potential dimensions contributing to Web QoE are portrayed)

Full size image

Our methodology, based on four phases, is illustrated in Fig. 2 and is inspired by guidelines provided by Bandara et al. [2] and Levy and Ellis [67]. We thus survey and compare different studies contributing to QoE modelling of Web browsing in each of these categories. One of the motives behind this approach is to provide a structured overview of findings, which can be utilized by interested parties (content providers, device manufacturers, network operators, researchers, and many more) depending on their interests and potential to utilize these findings in successful QoE management.

Fig. 2
figure 2

Methodology

Full size image

The analysis and comparison of papers in each of the categories is done according to the following parameters: addressed QoE IFs and the number of groups in which those IFs are varied, considered features/dimensions, type of Web task, and consideration in a mobile context. Furthermore, the comparisons address the study specific parameters such as type of study that has been conducted, sample size, used scale, conducted analysis, and manner of representing the resulting model. Finally, the analysis contains the key findings of each study.

We note again that quality perception is a multidimensional construct, and that being able to truly understand what impacts the QoE requires consideration of the interplay between different IFs, such as, for example, design, performance, and human-related factors. We thus move from studies considering isolated factors and user perceived dimensions to studies that adopt a multidimensional approach to QoE modelling. Once again, the motivation for such an approach is in providing the basis for future multidimensional QoE models for Web browsing, which are needed for successful optimization and management of QoE.

Given the multidisciplinary nature of the addressed topic, we surveyed papers coming from both the technical sciences domain (telecommunications, computer science, electrical engineering) and the social sciences. The methodology used in the starting phase for selecting the papers was the following: Due to a small number of studies actually quantifying Web QoE, our selection criteria has also included existing qualitative research studies addressing the impact of various IFs on QoE, which are mostly coming from the networking and telecommunications fields. Moreover, given that investigations from the HCI field have a similar goal directed towards a better understanding of the user experience and overall satisfaction when accessing Web sites and applications, our selection criteria has covered qualitative research studies addressing the impact of various IFs on the user experience. We thus searched relevant databases for papers with the following key words: web, browsing, mobile, QoE, user experience, aesthetics, usability, loading time, quality of information, modelling, user satisfaction. We have identified primary papers—ones that quantified QoE and UX, and secondary papers—ones that qualitatively described relations between impact factors and QoE and UX in the context of Web browsing. After going through retrieved papers, a total of 121 papers were selected to be included in the review given their relevance to the topic. 51.3% (62) of papers come from journals, 33% (40) are conference papers, while remaining 15.7% (19) of papers are book chapters, standards, dissertations, or Web sites. The distribution of considered papers per year, in the period from 2000 to 2017, is given in Fig. 3. Papers that proposed certain QoE related models (equation, statistical, or graphical representation) were considered and compared in tables in this paper (34 papers—28%). Other relevant papers (87 papers—72%) that have touched upon and qualitatively described the relations between IFs and QoE or relevant features/dimensions have been addressed in corresponding sections in the paper. The aim is to provide information on relations to be addressed in future multidimensional QoE modelling studies. In addition to scientific publications, we also give an overview of standardization efforts in the ITU related to subjective test methodologies and QoE factors related to Web browsing.

Fig. 3
figure 3

Number of considered papers per year in period 2000–2017

Full size image

We have structured the covered literature according to the study of the impact of different categories of QoE IFs that were varied or observed in the context of various perceived dimensions (dependent variables). The rest of the paper is organized as follows: section “System influence factors” deals with studies quantifying or addressing the system IFs’ impact on QoE, where we further distinguish between network-based temporal factors (for example, page loading times), and design factors (i.e. those related to aesthetics and usability). Section “Context IFs” addresses the impact of context IFs, while section “Human IFs” addresses research dealing with human-related IFs. In addition, the section “Key findings and lessons learned” summarises the key findings and lessons learned and elaborates on multidimensional approaches to QoE, that is, studies including multiple factors and aimed at modelling QoE in terms of multiple perceptual dimensions that contribute to the overall user quality judgement. Finally, section “Directions for future work: setting a research agenda” gives directions for future studies, thus providing a valuable reference for both practitioners and researchers in this field.

System influence factors

We adhere to the definition of system IFs as given by Le Callet, et al. [64] which refers to “the properties and characteristics that determine the technically produced quality of an application or service. They are related to media capture, coding, transmission, storage, rendering, and reproduction/display, as well as to the communication of information itself from content production to user.” These factors can be classified into four sub-categories:

  • network-related system IFs referring to data transmission over a network (bandwidth, delay, jitter, loss, throughput, etc.),

  • content-related system IFs referring to the content type and content reliability (specific temporal or spatial requirements, colour depth, texture, etc.),

  • device-related system IFs (display resolution, quality of touchscreen, etc.), and

  • media-related system IFs referring to media configuration factors (encoding, resolution, sampling rate, frame rate, etc.).

In the remainder of this section we will focus on those subcategories which are most relevant and have been mostly addressed in previous Web browsing research. Firstly, in section “Network-related IFs: waiting time” we focus on network-related IFs and review the relevant studies from the networking community that have addressed the user perception of waiting times in the context of Web QoE. Namely, in the past decade we have witnessed a rise in research focusing on the subjective end-user perspective when measuring the quality of services and underlying networks, as a shift beyond well-studied Quality of Service (QoS) concepts, which have focused primarily on the technical performance of systems (measurements of network delays, packet loss, etc.) as discussed by Schatz et al. [106]. Further on, section “Content and device-related IFs” looks at content- and device-related IFs, mostly addressed within the HCI field. Given the plethora of literature addressing the impact of design aspects such as aesthetics and usability on the user experience when accessing Web sites, we cite and discuss a number of relevant studies. Tables 1 and 2 summarise and compare existing literature in the fields of QoE modelling and UX for Web browsing literature according to the adopted methodology. The tables contain the key findings of each study, the details of which are discussed in following sections. Finally, given its relevance to Web QoE, in the context of content-related IFs we also discuss the impact of the quality of information presented on a Web site (section “Quality of information”).

Table 1 Survey of selected existing QoE related modelling approaches considering temporal aspects in the context of Web browsing
Full size table
Table 2 Survey of selected existing HCI-related modelling approaches considering aesthetics and usability in the context of Web browsing
Full size table

Network-related IFs: waiting time

Existing studies in the field of QoE modelling of Web browsing (Table 1) argue that waiting time is the key influencing factor when it comes to end-user Web QoE [1, 28, 29, 33, 45, 46, 83, 84, 93, 103, 105]. Therefore, the subjective perception of waiting time resulting from waiting for Web pages to load is considered to be the key QoE feature, while page load time (PLT) has been characterized to be the most important QoE IF. PLT has been defined as “the time elapsed between the URL-request (for example, click on the link) and the finished rendering of the Web page” [29, 51], and is in turn influenced by multiple system features (for example, downlink bandwidth or delay as addressed by ITU-T Recommendation G.1030 [51]). In other words, the longer we have to wait for the Web page to load (or transactions to complete), the more dissatisfied we tend to become (as discussed by Egger et al. [29]). It should be noted that recent moves to HTTP/2, such as studied by Bocchi et al. [136], are pushing to optimize the Web by decreasing PLT using (amongst other features) a server push mechanism. We note that the issue of loading times has also been addressed in the context of certain Web usability studies such as Flavian et al. [31] and Palmer [86]. In a study addressing the influence of perceived usability on users’ loyalty to Web sites, Flavian et al. [31] consider perceived loading speed to be a measure related to perceived usability.

The authors of the existing Web-related QoE experiments and studies have described the relations between QoE, perceived waiting time, and corresponding factors by proposing concrete mathematical models [28, 50, 51, 83, 84, 93], using graphical representation [28, 29, 105], or proving strong correlations by using statistical analysis tools [83, 84]. Also, they have identified logarithmic relationships between QoE and waiting/response times [28, 46, 50, 51, 93].

A standardized description of QoE and temporal aspects, which was given in ITU-T Recommendation G.1030 [50] (updated in ITU-T Recommendation G.1031 [51]), aims to establish the relation between users’ responses and duration of Web browsing session time which resulted in strong correlations of the results showing that user perceived quality decreases linearly with the logarithm of the session time. The update of the standard focuses primarily on assessing the dependency of user perceived quality on network performance metrics (mapping HTTP response and download times to perceived quality of a Web browsing session). Also, this relationship has been explained in the context of the well know Weber-Fechner Law (WFL) [134] which (based on human perceptive abilities) states that the just-noticeable difference between two levels of a certain stimulus is proportional to the magnitude of the stimuli (in this case referring to waiting time). Reichl et al. [93] have described the relationship between QoE and PLT for a simple Web browsing scenario (online photo album) in the spirit of the abovementioned WFL principle. However, in complex Web browsing scenarios (for example, informative Web site), the authors claim that the WFL holds up to a certain extent, that is, the Mean Opinion Score (MOS) increases logarithmically with increasing downlink bandwidth, due to decreasing PLT, but stagnates in cases of high download bandwidth. This phenomenon has been explained as the result of nonlinear mapping of PLTs to bandwidth and the impact of user and context factors.

Similarly, Egger et al. [28] have formulated the WQL hypothesis (the relationship between Waiting time and its QoE evaluation on a linear Absolute Category Rating (ACR) scale is Logarithmic) claiming that “the relationship between waiting time and its QoE evaluation on a linear ACR scale is logarithmic”. In addition, Varela et al. [121, 122] have examined the combined effect of aesthetics and waiting times on QoE at the same time.

Further on, as discussed by Baraković and Skorin-Kapov [4] the impact of waiting times is an important issue in the case of Web content accessed via wireless networks, where unreliable and variable conditions of wireless channels, network conditions, signalling traffic overload, bandwidth limitations, and frequent bandwidth variations may result in significant waiting times. As previously stated, PLT is defined as the time elapsed between the URL-request (for example, click on the link) and the finished rendering of the Web page. Consequently, Web sites adapted for mobile access take into account not only design aspects (for example, responsive Web design [75], but aim to reduce loading times (for example, including fewer media components such as pictures or videos, or those of lower resolution) [32], since today users use their mobile devices in different contexts and environments, with increased performance expectations leading to decreased delay tolerance. There exist numerous Internet posts giving suggestions and advice on how to decrease waiting time in the mobile Web context, while several studies have addressed the impact of waiting time on user experience in the mobile context [5, 47, 59, 95, 103, 104, 116, 133].

Recently, Ickin et al. [47] addressed the factors influencing QoE, including interface design, application performance, battery efficiency, mobile device features, application and connectivity cost, user routine and lifestyle. The authors showed how MOS levels differ in dependence of various QoS parameters, namely round trip time, service response time, and throughput (we note that a five-point ACR scale used to obtain MOS values as specified in ITU-T Recommendation P.800.1. is very commonly used in subjective studies as a “simplified” measure of QoE). In other words, the increased values of the first two parameters (for example, round trip time or server response time raising from 200 to 1400 ms) and decreased value of the throughput (for example throughput falling from 20 kbytes/s to 0) correspond to lower MOS values (going from 5 to 2 or 1), and vice versa. On the other hand, Schatz and Egger in their Web browsing study reported in [104] with smartphones and tablets have come to the conclusion that the network QoS together with the media type is not often the main problem for users, but the mobile device characteristics. It was concluded that the loading time of a mobile Web site should be low in order to compensate for the low screen resolution and limited processing speed. Further on, Baraković and Skorin-Kapov [5] have quantified and modelled the impact of waiting time on mobile Web browsing QoE in a multidimensional fashion considering other dimensions as well (presented and discussed later in the paper) for several scenarios differing in used mobile device (smartphone and tablet) and accessed Web content (news portal, thematic portal, e-mail). The authors showed that Web site loading time influences mobile Web QoE, but when tested within certain bounds (loading times no longer than 8 s) was found to not be the most important influencing factor (rather, design factors such as aesthetics and usability had stronger impacts on QoE). This can provide insight in the telecommunications and networking field in terms of justifying the need to consider additional Web site design-related factors (beyond only loading times) when studying overall user experience. Also, in their extended study given in [133] they have proved that the number of taps impact the perception of Web site loading time.

Given that Web browsing is much more than pure Web page loading, it is a rather highly interactive and complex activity which is consequently affected by a wide spectrum of system, user, and context factors. Not questioning the high importance of temporal aspects in Web QoE modelling, a number of studies also stress the importance of examining Web QoE on a consecutive series of page views under the constraints of a given task for a user, and suggest inclusion of novel metrics and factors in the Web QoE modelling process in order to perform the action reliably and with respect to the true interactive nature of Web browsing. As previously mentioned, ITU-T Rec. P.1501 highlights the definition of a Web browsing session, whereby a key issue is that during a browsing session users see/perceive a large number of different PLT’s. Consequently, there is a need to consider the impact of a given distribution of PLT’s on QoE. However, to-date research results have not yet provided a model of how this distribution should be considered.

Recent studies going beyond single page loading times have built on earlier findings, such as Bhatti et al. [9], and also taken into account users’ tasks, content of the Web site, per-element loading times, and loading strategy as part of the QoE assessment process. The results reported by Strohmeier et al. [113] show that the impact of per-element loading times (referring to Web page elements such as pictures, banners, etc.) and loading order are dependent on the type of task a user is engaged in (free exploration vs. search tasks). This work has been complemented with the study of Guse et al. [37] which shows that the failure to load elements leads to severely lower quality ratings. However, when it comes to todays’ Web pages, that are mostly built by using HTML5/Ajax technology, users do not perceive whole loading times, but rather sense only element refreshes, which represents an issue that goes beyond the work of Strohmeier et al. [113] and Guse et al. [37]. Therefore, this influence should be addressed as major challenge in future research in the field of Web QoE modelling. Also, temporal related throughput fluctuations and their impact on user Web QoE were examined by Sackl et al. in [98, 102] for different Web applications. The results showed that subjective perception of network fluctuations strongly depends on the Web application (for example, users are more tolerant regarding fluctuations while uploading a file in contrast to browsing an online photo album).

The key findings reported in this section are summarised in section “Key findings and lessons learned”. Although the aforementioned studies focus on the impacts of waiting times on users’ quality ratings, it is clear that additional factors need to be considered (in a multidimensional fashion) if aiming for a more holistic understanding of QoE (as reflected also in ITU-T Recommendation G.1031 [51]).

Content and device-related IFs

Additional challenges in the field of modelling QoE for Web browsing are driven by the fact that Web sites are becoming increasingly complex in terms of users’ expectations related to visual appeal, usability, interactivity, technologies used for development, devices used to access, etc. (as given in ITU-T recommendation G.1031 [51]). Reported user studies from the networking and telecommunications fields, characterized by their focus on temporal aspects, for the most part disregard the impact of additional Web content design aspects, namely those related to aesthetics or usability. Such aspects are important in the field of QoE modelling, and have to a great extent been studied in the domain of UX research.

Factors impacting perceived aesthetics

One of two most extensively studied dimensions in UX research has been the aesthetics of user interfaces [7, 23, 38, 63, 65, 76, 81, 87, 97, 117, 118, 120], which consists of two main dimensions: classical aesthetics (characterized by the terms “aesthetic”, “cleanness”, “clearness”, “symmetry”, etc.) and expressive aesthetics (characterized by the terms “original”, “creative”, “fascinating”). The aforementioned research studies suggest that aesthetics serves a major role in affecting the user experience when viewing and using a Web site.

Some authors addressing the field of Web aesthetics have considered aesthetics as a composite factor construct composed of its constituent parts such as colour, visual layout, typography, symmetry, saturation, brightness, etc. [5, 65, 76, 97, 117, 118, 121, 122, 133], while others considered and investigated aesthetics in terms of its various previously mentioned constituent parts [24, 38, 63, 81, 108, 120]. In all cases, the impact of Web site visual appeal on user satisfaction has been proven. For example, with regard to colour related research, several studies have shown that the colour of Web sites has great influence on user perception such that if the colour appeal is higher, the user satisfaction in various contexts increases [17, 24, 38, 60].

A study that has dealt with the investigation of mobile Web site design is the one carried out by Cyr et al. [23]. Namely, they have shown that regardless of technology and device, interface design matters, which in other words means that the aesthetics of a mobile Web site impacts user satisfaction. In addition, the aforementioned research has shown interesting relations among gender, age, culture, and three dimensions of Web design (specified below) defined by Garrett [34]. Also, according to Reichensten [92], the commonly used black and white combination is not suitable for contemporary high contrast screens of mobile devices, although in Web design it is sometimes considered preferable (as discussed by Blue et al. [10]), while somewhere less aesthetic as noted by Hall and Hanna [38] and Lee and Koubek [65]. Also, Reichensten [92] recommends the usage of dark grey for text and light grey for the background.

Very often, the aesthetics of a Web site (whether viewed via a desktop screen or on a mobile device) is intermixed with the notion of design. There exists a whole range of recommendations and tips for designing Web sites, but these texts go beyond visual appeal. In that context, and since it is relevant for this review, Garrett in [34] has suggested Web design categories for:

  • visual design (related to balance, emotional appeal, aesthetics, and uniformity of the overall graphical look of a Web site, which includes colour, shapes, typography, images, etc.),

  • navigational design (related to Web site features that help or hinder users during interaction), and

  • information design (refers to clearness, logical representation, quality of information provided by Web site, etc.).

Therefore, Web design may refer to the area where aesthetics overlap with other important features that impact QoE and user satisfaction with a given Web site, namely those related to usability.

Factors impacting perceived usability

Web site usability is another extensively studied concept, which is broad in nature and has been defined as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in specified context of use” by ISO [48]. While satisfaction is considered to be a key element of the user experience as discussed by Borsci et al. [12], Hassenzahl [41], and Lewis [68], it may also be incorporated in the meaning of QoE considering the notion of “delight”, together with the notions of pleasure, functionality, excitement, etc. which constitute the user experience. Kefalidou et al. [56] also discuss and suggest that “delight” is something more than satisfaction and recognizes the need for the HCI community concentrated on UX to address “delight” as such and explore the appropriate methods to assess it.

Studies examining the effect of perceived usability on experiences start with the initial finding that a product’s visual appeal influences its perceived usability, as discussed by Tractinsky et al. [115], and they suggest that good perceived usability results in positive reactions and experiences, while poor perception leads to negative ones [71, 72, 89, 114]. While perceived usability is an outcome which describes the match between user, task, and system in a context of use, in the following text we focus primarily on system related factors.

When it comes to usability in a mobile Web browsing context, the unique features of mobile devices and wireless networks are found to pose a number of significant challenges for examining the user experience. Besides the numerous advantages of mobile devices, literature mostly focuses on constraints of the corresponding hardware and basic functionalities. Firstly, there is a mobile device’s screen size that affects the user experience when browsing the Web and limits the interactivity. This factor has been addressed by Coursaris and Kim [19], Harrison et al. [39], Lobo et al. [69], Maniar et al. [73], Marcial [74], Raptis et al. [91], Roto [95], Schatz and Egger [104], Schleicher et al. [107], Wroblewski [126], and Zhang et al. [128]. However, the issue of screen size may be addressed—as any other mobility related issue, in terms of two widely used device types, smartphones and tablets—which are often used in different contexts and present two different device groups. Namely, tablets are overwhelmingly used at home and for “lazy Internet usage”, that is, consuming media and content, as well as browsing. Smartphones, although being used at home for a high 40% of time, are still most associated with being on the go and are used primarily for communication, content snacking, and mobile apps usage as discussed by Baraković [3]. Due to smartphones being used outdoors and on the go more often, there is a massive difference in the types of experiences people expect based on the device they are using. Therefore, one should approach this topic very carefully. For example, Schatz and Egger [104] have shown that tablets perform better in comparison to considered smartphones due to a larger screen, but neglected the fact that often times these devices are used in different contexts.

Secondly, driven by the desire to maximize the screen and avoid physical controls (mouse, keyboard, etc.), touch-sensitive displays have become widespread [107]. Now, the plane used for output is used for input as well. Touchscreens that brought additional usability challenges in the interaction domain (for example, “fat finger problem”) have been addressed by Brangdon et al. [13], Schleicher et al. [107], Siegenthaler et al. [109]. Further on, touchscreens have contributed to Web page navigation in the mobile context. Several studies such as Burigat et al. [14], Isomursu et al. [52], Kaasinen et al. [55], Roto [95], Siegenthaler et al. [109], and Vartiainen et al. [124] addressed this issue concluding that the navigation is quite important, and different navigational patterns may contribute to usability while browsing the mobile Web. In addition, an interesting finding revealed by Diefenbach and Hassenzahl in [26], found that product beauty was valuated, but discounted when choosing between a beautiful and a usable mobile device.

The advance of mobile device technology brings mobile devices with much larger memory, CPU resources or battery lifetime than before, but it seems that users are using more as well and they are encouraged to demand more due to the fact that stakeholders in the service provisioning chain (device manufacturers, network operators, etc.) tend to be user-oriented. Given the fact that today’s Web browsing activities include multimedia content as well (which is much more power consuming, for example, video streaming), these features may present bottlenecks and barriers to a good user experience in terms of battery consumption [11, 111].

Finally, given the challenges related to mobile usability in general, Coursaris and Kim [19] gave a comprehensive and detailed meta-analytical review of more than 100 empirical mobile usability studies (only a few of them related to mobile Web browsing). The recommendations of the subject review are general, but some of those that apply to mobile Web browsing are: (1) consider the wide range of usability dimensions identified in this study when evaluating the usability of mobile interfaces and applications; (2) design mobile interfaces and applications that fit particular settings, while being flexible to accommodate others; and (3) explore the human factors and interplay among all the addressed dynamic factors in terms of their impact on mobile usability.

So far, the predominant research approach in mobile HCI appears to be explorative or data-driven. In other words, data is collected for the purpose of better understanding and improving mobile HCI, and clear research questions and hypotheses derived from the theory are used as a starting point. If done so, it is rather the application of an existing model of human behaviour to mobile interaction. Moreover, the majority of existing mobile HCI publications devote more space to how people use their devices or services, and less to why they do so [107]. In other words, existing HCI studies that actually investigate the impact of the abovementioned aesthetic or usability characteristics or aesthetics or usability as a factor impacting UX in the mobile Web context are descriptive, while the ones that quantify (by equation, or statistical, or graphical representation) or model the relations, that is, QoE studies, are limited.

Relations between aesthetics and usability

Researchers such as Ben-Bassat et al. [8], Hartmann et al. [40], Hassenzahl and Monk [42], Kurosu and Kashimura [61], Lee and Koubek [65], Moshagen et al. [80], Sonderegger et al. [110], or Tuch et al. [118] have extensively studied the relations between aesthetics and usability of Web sites, in particular referring to the relationships between perceived usability, perceived aesthetics, and overall user preference in Web site interaction. Tuch et al. in [118] further provide a summary of both correlative and experimental studies that have investigated the aesthetic-usability relation. Correlative studies are the ones that give models based on correlative data, that is, the causality is solely a matter of theoretical reasoning and cannot be tested by existing data, while experimental studies give models which are based on data obtained when manipulating certain factors as independent variables. Motivated by both the limited number of experimental studies (as opposed to a greater number of correlative studies) investigating this relation and the variability in the findings, Tuch et al. report on an experimental study involving usability and aesthetic manipulations in the case of a Web shop [118]. They found that while aesthetics did not affect usability, usability had an effect on post-use perceived aesthetics. Other experimental studies such as Ben-Bassat et al. [8], Kurosu and Kashimura [61], or Lee and Koubek [65] have shown a significant effect of aesthetics on perceived usability, while Lee and Koubek [65] also report evidence on the effect of usability on perceived aesthetics. In addition, Sonderegger et al. [110] have shown that although aesthetics had a positive impact on perceived usability, it began to wane with increasing exposure time. However, the interested reader is further referred to a comprehensive review reported by Lee and Koubek [66], where the authors compare the relationships among perceived usability, perceived aesthetics, user performance and user preference, with the consideration of occurrence of actual use (before and during/after actual use).

Aesthetics and usability in Web QoE studies

When looking to explain the way aesthetics and usability may affect Web QoE, we refer to the widely accepted definition of QoE given by Le Callet et al. [64], that is, QoE as being the “degree of delight or annoyance”. According to Oxford dictionaries [85] and the discussion given by Kefalidou et al. [56], “delight” is a “great pleasure” or “a cause or source of great pleasure”, while one of possible explanations of “annoyance” is “displeasure”. Hence, given the fact that aesthetics and usability impact pleasure, they impact the degree of delight or annoyance, that is, QoE. Therefore, it is clear that both aesthetics and usability aspects (influenced by a number of user, system, and context factors) which affect pleasure, should be considered when aiming to provide a holistic approach to understanding QoE, given that pleasure is according to Kurosu [62] and through Kansei aspects a main part of UX interaction.

In addition, with QoE having been defined as the “degree of delight or annoyance of the user of an application or service” resulting “from the fulfilment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the user’s personality and current state”, it is clear that both hedonic qualities (linked to aesthetics and the ability to evoke pleasure) and pragmatic qualities (linked to usability and the ability to fulfil user needs) [41] contribute to the degree of delight or annoyance experienced by the user. In their taxonomy of QoE and QoS aspects of multimodal human–machine interaction, Möller et al. [79] further relate QoE aspects (including both hedonic and pragmatic components) to the service quality perception and judgement processes. Further on, savvy users are often no longer “delighted” with simply average Web site design that loads quickly, but expect a site to meet their (device/network) capabilities, preferences, and needs. In other words, they want it to fulfil their high expectations with respect to both hedonic and/or pragmatic quality, hence linked to the overall QoE judgement of a Web browsing session [5].

In the mobile Web and Web QoE context, aesthetics and usability factors have been investigated by Baraković and Skorin-Kapov [5, 133] and Varela et al. [122], respectively. The former study [5], addressing mobile Web QoE browsing in a multidimensional fashion, showed a strong and positive impact of the usability and aesthetics perception on QoE. The study revealed, through quantification and modelling of relations between QoE and considered QoE dimensions (described in section “Human IFs” dealing with multidimensional modelling of QoE), that the perception of Web site usability and aesthetics have a stronger impact on QoE than waiting times (assuming waiting times are not longer than 8 s). Additionally, the study given in [133] (which is also describe in detail in section “Human IFs”) showed that there exists negative impact of Web site loading time and positive impact of aesthetics and quality of information on perceived Web site usability. Also, it showed that number of taps to reach the desired content on the Web has negative impact on perceived Web site aesthetics, while quality of information has a positive one. The latter study [122] explored the impact of visual appeal on desktop Web QoE and showed that the overall QoE is strongly correlated with perceived aesthetics, perceived ease-of-use, and perceived network performance. As discussed by Porat and Tractinsky [87], recent product development and empirical studies confirm that the aesthetics aspects of various computing products serve an important role in shaping user attitudes and emotional states in general, particularly in the context of Web. The beauty of IT products is perceived as a hedonic attribute of the product, which leads to pleasurable user experience and to increased satisfaction in users. Further on, they have also discussed that better usability is likely to increase pleasure, whereas lower levels of usability increase frustration, and thus reduce pleasure. The aforementioned authors proved that aesthetics and usability levels contribute to and increase user’s pleasure in the Web context. Therefore, the aesthetics and usability of Web sites are a strong determinant of pleasure experienced by the user during the interaction with a given service (in this context mobile Web site) as it is discussed by Hassenzahl [41], Lavie and Tractinsky [63], and Porat and Tractinsky [87].

Quality of information

The essence of many Web browsing activities is in obtaining desired information, whose quality thereof can increase or decrease the delight and pleasure, or on the other hand annoyance with the Web site, and thereby directly influence QoE. McKinney et al. [78] have defined Web-based information quality as the “user’s perception of the quality of information presented on a Web site.” Yang et al. [127] have classified dimensions of information quality into usefulness of content and adequacy of information. The first group refers to the value (relevancy and clearness), reliability (accuracy, dependability, and consistency), currency (timeliness and continuous update), and accuracy (degree to which the system information is free of error) of information. The second group is the extent of completeness of information, since Web sites need to provide information to facilitate user’s content understanding. The authors of this study (involving 1992 participants) have shown that this factor has a high impact on user satisfaction with the overall quality of a Web site, and that users demand unique, reliable, valuable, and up-to-date information on Web sites. In the mobile Web browsing context, Baraković and Skorin-Kapov in [5] provided models with quality of information as a predictor for user Web site QoE and proved its effect, as it will be discussed further in the paper. The same authors also proved in their work given in [133] that Web site aesthetics has a positive impact on perceived quality of Web site information.

In today’s environment where users are exposed to a plethora of so-called “junk information”, the abovementioned demands clearly make sense, since if the required information is not of good quality or the users’ perception of quality is poor, it may cause (or increase) annoyance with the whole Web site. On the other hand, well-structured information may have a positive impact on the ability of a user to complete a desired task (finding desired information, completing a Web-based transaction, etc.), or on the overall aesthetic appeal of a Web site leading to pleasure and enjoyment of use, since users are oriented towards information finding and their subjective look-and-feel of portal design [5, 35]. However, presenting information on mobile Web sites is much more challenging as compared to desktop Web sites, since mobile users even more scan the content hunting for information they are after rather than reading in details (as presented by Lambrea [135]).

Existing literature has recognized that information quality presented on mobile Web sites contributes to user experience [15, 23, 77], while the degree of the impact depends on the goals of Web browsing [113]. The main constraints are the limited screen size and usage context, as discussed by Jeong and Han [53], leading to numerous approaches, recommendations, guidelines, and tips for effective writing (concise, short, simple, etc.). However, although most of the existing literature propagates content adjustment, McGrane [77] claims that content should not be tailored exclusively for people who are on the move and require only important information. Namely, users do not have to browse the Web and read the information via their limited screen size smartphones while moving or being in a hurry. They may also do that via other mobile devices such as tablets that have larger screens compared to smartphones, or in different context, that is, via their smartphones in stationary conditions (for example, lying in bed on Sunday morning).

Context IFs

In addition to the most common factors found in the literature, this section and the following one address existing literature considering the wide range of context and human IFs in the context of QoE (and UX) for Web browsing. The number of these factors is large and their disparity high. While it is clearly impossible to consider all possible factors that may have an impact on QoE, those that have been identified as relevant in surveyed studies are highlighted. In addition, Table 3 summarises selected existing literature in the fields of QoE and UX modelling for Web browsing.

Table 3 Survey of selected existing QoE and UX-related modelling approaches considering human and context IFs in the context of Web browsing
Full size table
Table 4 Survey of selected existing QoE related multidimensional modelling approaches in the context of Web browsing
Full size table

Context IFs are “factors that embrace any situational property to describe the user’s environment in terms of physical, temporal, social, economic, task and technical characteristics” (in Le Callet et al. [64], as taken from [54]). They can be categorized as proposed by Jumisko-Pyykkö and Vainio in [54] into the following groups:

  • physical (characteristics of location and space (including movement within and transactions between locations), spatial location, functional place and space, sensed environmental attributes, movements and mobility),

  • temporal (characteristics such as time of day, week, month, season, year, duration, frequency of use of service, etc.),

  • social (characterized by interpersonal relations existing during the experience (alone or with other persons)),

  • economic (for example, cost, subscription type, or brand of the application, i.e., service),

  • task (QoE towards a given service may be as well influenced by the type of task that users execute), and

  • technical and information factors (related to the relationship between the application of interest and other relevant services).

We note that in the following paragraphs we address only a subset of these factors based on results found in surveyed papers.

Physical

According to the empirical study reported by Guse et al. [36], aimed at determining the impact of environmental distractions (for example, other people or traffic) on mobile Web browsing QoE, results showed that QoE ratings were not affected by the considered context or distractions (in this case browsing while using public transport vs. sitting at home on the sofa), while on the other hand the browsing task itself had a significant impact. A conclusion that may be drawn is that there is a need for more studies conducted in “real-world” settings as opposed to controlled laboratory environments so as to further study the impact of various environmental and situational factors.

Temporal

In the context of user expectations and prior experience, Hoßfeld et al. [44] studied the impact of the so called memory effect on Web browsing QoE. The previously discussed Web QoE modelling approaches consider temporality in the form of waiting times and focus on the current stimuli, that is, the actual service environment and conditions, thereby neglecting the impact of the temporal dynamics or historical experiences of the user satisfaction while consuming a certain service, that is, the memory effect. Hoßfeld et al. [44] found that, although the current QoS level clearly determines resulting end-user quality ratings, there is also a visible influence of the quality levels experienced in previous test conditions. In particular, in addition to the current QoS level the user experienced, the technical QoS parameters of the preceding browsing session have to be taken into account. The contributions of the paper are twofold: it requires the quantification of the memory effect’s impact on QoE (the consideration of Web sessions rather than single Web pages); and time-dynamics and the internal state of the user are essential components of the Web experience and need to be adequately reflected in Web QoE models.

Economic

Finally, an important factor affecting both user satisfaction and expectations is the price of the service, that is, in this case the cost that a (mobile) user is paying for a Web browsing service. Today, costs in this context are usually related to network/connectivity (for example, users expect to wait less for Web pages to load if they have paid more, hence implying a faster network connection) and device (PC/laptop/smartphone/tablet) performance (for example, if users paid more money for a certain device they inherently expect better performance). In other words, users want benefits to exceed the outlays. Rarely is the price mentioned in the context of aesthetics, usability, or provided information, but there are situations where certain Web portals require payment in order to access their content (for example, online newspapers).

However, this does not mean that price does not have an effect on previously mentioned features (perceived aesthetics, usability, delay, etc.) that impact a user experience due to different perceptions of costs. Namely, different price values affect users differently. In other words, a user satisfaction towards a certain service is affected by price awareness, as discussed by Evanschitzky et al. [30], Uddin and Akhter [119], and Varki and Colgate [123], and perceived value obtained for a given price, with higher perceived value clearly resulting in higher user satisfaction. The price awareness and perception in the context of various services have been of major interest in the fields of marketing management and economics, while research done by Reichl et al. [93] and Sackl et al. [101] contribute to establishment of a link between willingness to pay (WTP) and its impact on QoE for video watching. The results of studies have shown that the connection between spending money and quality perception is quite unclear and further research activities are needed in this domain. In fact, one needs to ask if it is even possible to achieve clear results in contrived test situations given that the user does not necessarily feel, act, or perceive the service in the same way when taking part in test scenarios versus real world scenarios involving actual payments. In these situations, the background emotions and thoughts of the user are different and need to be accounted for and better understood. This further imposes questions as to what extent it is possible to obtain reliable data for examining the impact of WTP on QoE in experiments which do not include real-life transactions. This impact, of course, needs to be further addressed by the research community. Further on, according to the best of the authors’ knowledge, there exist no studies that study the connection between price and QoE for Web or mobile Web browsing, although a study reported by Baraković [3] showed that users were very concerned about incurred costs when browsing over a mobile network. However, further research is needed to ascertain the relationships between price, QoE, and a users’ willingness to pay for a certain “quality level”, which in the context of Web QoE may be considered, for example, in terms of network speed (impacting waiting times) and/or the quality of (or amount of) information available on a given Web site.

Task

As previously mentioned in relation to Web QoE studies focusing on the impact of waiting times, the notion of user task has been considered as an important contextual IF. Namely, as recognized by Bhatti, et al. [9], Guse, et al. [36], and Strohmeier, et al. [113], a users’ goal in interacting with a given Web site, that is, the task that the user performs while browsing the Web, may influence his or her overall satisfaction with the service due to many reasons such as:

  • the nature of the task:

    • hedonic: for example, reading online news portal or communication with friends,

    • functional: for example, searching for information related to business or business communication,

  • user’s goal fulfilment in relation to the subject task:

    • successful: for example, information found,

    • unsuccessful: for example, information not found,

  • duration of the task:

    • time-consuming: for example, user interacts with the Web site longer than he or she expects,

    • non-time-consuming: for example, user interacts with the Web site shorter than he or she expects.

Furthermore, the difference between before and after actual task performance, that is, time when the Web browsing occurred, has been shown to be a significant consideration with respect to perceived usability, perceived aesthetics, and user preferences. As reported by Lee and Koubek [65] and Tuch et al. [118], a user preference before Web browsing is affected by aesthetics, while after use by both aesthetics and usability.

To summarize, there are many context IFs that impact QoE in the Web browsing context. Some of those, such as previously discussed price, task, memory effect, etc. have been addressed by the research community. However, it is necessary for researchers to extend the scope of studies and to properly investigate the impact of a larger number of aforementioned context IFs on QoE in Web and mobile Web environments. The aim would be to identify the key factors relevant for understanding and managing the overall QoE.

Human IFs

Human IFs present “any variant or invariant property or characteristic of a human user. The characteristic can describe the demographic and socio-economic background, the physical and mental constitution, or the user’s emotional state”, as given by Le Callet et al. [64]. They are highly complex because of their subjectivity and relation to internal states and processes. In addition, they are strongly interrelated and may also significantly interplay with other groups of IFs. These factors are divided into two sub-categories:

  • low-level processing IFs related to the physical, emotional, and mental constitution (e.g. gender, age, lower-order emotions, user mood, personality traits, motivation, attention level, etc.), and

  • higher-level cognitive processing IFs (socio-economic situation, educational background, attitudes and values, expectations, needs, knowledge, previous experiences, etc.).

These factors have been considered to a limited extent in the context of QoE, and due to lack of empirical evidence, they still remain not well comprehended. However, it needs to be noted that there is a long and well established discipline of ergonomics and human factors whose survey is out of scope of this paper.

In a study given in [33] (besides recognizing the importance of waiting times in the mobile context in terms of user perceived quality), Galletta et al. have also included the impact of context IFs (familiarity and organization (visual layout and number of clicks) of Web site) and concluded that the negative impacts of delay, that is, temporal factor (noted earlier), are the strongest when delays are longer than expected, or if they occur in unpredictable patterns. A similar conclusion related to user expectations as well as his or her prior experiences has been drawn by Ibarrola et al. [46]. Namely, this study has shown that expert users become frustrated with delays occurring during the experiment (earlier than non-expert users) because of their higher demanding expectations. In addition, unskilled users became more critical in the repeated experiment. Despite the strong consensus within the QoE community about the heavy influence of user expectations on quality perception and judgment, the actual integration of this factor in QoE modelling has been hitherto neglected. Sackl and Schatz [99, 100] demonstrated in their studies that including data about individual user expectations in QoE assessment can significantly increase MOS prediction accuracy of resulting models.

As previously mentioned and according to reviewed literature, additional user-oriented factors that affect Web and mobile Web site perception are culture, gender, and age. Namely, according to the existing literature such as Barber and Badre [6], Cyr et al. [23–25], or Dong and Lee [27], user preferences for Web site design and its attributes are known to vary across different cultures. Culture also affects Web site information perception, since it is after all very subjective in nature. On the other hand, regardless of culture, users prefer to navigate Web sites easily. However, there are rare studies that investigate the impact of culture in the mobile Web site context such as one performed by Cyr et al. [22].

Further on, existing studies that examined the relation of gender and Web site perception such as Chen and Dhillon [16] and Cyr et al. [21] have shown that there are differences in how men and women perceive various dimensions of Web sites. For example, women more than men desire detailed information content and they are more interested in visual beauty and aesthetics, while men are more concerned with visual ease, that is, usability of Web sites as discussed by Cyr et al. [23] and Web site symmetry as discussed by Tuch et al. [117]. On the other hand, Coursaris et al. [20] claim that gender does not affect Web site preferences. Clearly additional research is needed to draw reliable conclusions.

Age is another common dimension frequently examined in the HCI context that affects user satisfaction. Namely, elderly people and younger people perceive, comprehend and use technology, and thereby Web and mobile Web browsing, in different manners and contexts. Age may be related to physical limitations such as wearing glasses (typing and small screens then become a difficulty), or willingness to accept new trends which may cause different tolerance thresholds towards certain services or products. Therefore, it may be concluded that the impact of age is significant, since it moderates user perception in various manners. For example, age groups may differ in their tolerance to waiting time, opinions regarding design aesthetics, perception of system usability, etc. Cyr et al. [23] claim that mobile Web site perception is affected by age, that is., significant differences in satisfaction with mobile Web site were found between older and younger subject groups. On the other hand, research results given by Baraković [3] show that age per se does not affect user perception, but other user and context factors that differ across different age groups, such as previous experience with a given technology, visual difficulties, users’ character, culture, etc. However, to draw relevant conclusions, it is clear that further research is needed regarding the impact of age on Web browsing QoE.

As concluded in the previous section, there are a number of human IFs that affect QoE. Some, such as age, previous expectations, gender, etc. have been addressed by the research community, as stated above. An ongoing challenge for QoE researchers is to extend the scope of studies and include disciplines of ergonomics and human factors. In other word, there is a need to properly investigate the impact of a wider scope of human IFs on Web browsing QoE, so as to identify the ones that are crucial in estimating QoE and delivering personalized services.

Key findings and lessons learned

Towards multidimensional models

Based on previously elaborated findings, it may be concluded that QoE in the context of Web browsing is a fast emerging multidisciplinary field based on social psychology, cognitive science, economics, and engineering science, focused on understanding overall human quality requirements [4]. In particular, addressed studies, as well as ITU-T Recommendation G.1031 [51], show that there is a wide range of factors and features affecting the QoE of Web and mobile Web browsing, and that QoE should be considered in a multidimensional fashion if aiming for a more holistic understanding and approach. Prior research has generally neglected to address the simultaneous impacts of various factors and dimensions on the quality of the overall user experience in the considered context, which can be justified by the complexity that such studies incur. However, a limited number of studies such as Baraković and Skorin-Kapov [5, 133] and Nguyen et al. [83], Strohmeier et al. [113], or Varela et al. [122], have considered the simultaneous effect of multiple factors and dimensions on Web browsing in their investigation (Fig. 4; Table 1), while there are even less approaches that have actually quantified and modelled the relations, such as Baraković and Skorin-Kapov [5, 133]. However, these studies do not include human factors, which is a general drawback of studies conducted in the QoE field as compared to HCI studies.

Fig. 4
figure 4

Overview of chosen multidimensional QoE modelling related studies (Grey arrows represent examined impacts, while black and bolded arrows represent confirmed effects): a Varela et al. [122]; b Nguyen et al. [83] and Strohmeier et al. [113]; c Baraković and Skorin-Kapov [5, 133]

Full size image

In a stationary/desktop Web context, Varela et al. in [122] used a multidimensional approach to investigate Web QoE by focusing on studying three key dimensions as contributing to overall Web QoE: perceived performance (in terms of page loading time), perceived aesthetics, and perceived ease-of-use (considered as a sub-dimension of usability) for a news site, a photo sharing application, and an e-commerce site. Key results have shown that PLT and visual appeal have a significant effect on overall user QoE and that both, higher perceived aesthetics and ease-of-use, results in increased users’ tolerance to delay. Also, the research proved that there exists strong correlation between overall QoE and perceived aesthetics, ease-of-use, and network performance.

In the mobile Web browsing context (browsing information, thematic, and e-mail portals via both a smartphone and tablet), Baraković and Skorin-Kapov [5] have proposed multidimensional models that represent and quantify mutual relations of QoE and key features, that is, perceived Web site loading time, perceived aesthetics of Web site, perceived usability of Web sites, and perceived quality of Web site information. The contribution of this research is three-fold. Firstly, QoE in a mobile Web browsing context is addressed as a multidimensional concept. Secondly, the authors have shown that the impact of PLT, aesthetics, usability, and quality of information provided by Web sites on mobile Web QoE exists. Thirdly, mutual relations between QoE and its features are quantified, and based on the obtained models, one is able to identify the importance (impact degree) of distinct dimensions in terms of considered perceptions and overall QoE. Therefore, the perception of Web site usability, aesthetics, loading time, and quality of information respectively in that order differ in the degree to which they impact the overall QoE (going from most to least influential) regardless of performed task or used device in a mobile Web browsing context. In other words, the multidimensional models for mobile Web browsing QoE show that the most important perceptual dimensions were found to be perceived Web site usability and aesthetics, respectively, and that they impact QoE in a mobile environment more than the perception of Web site loading time, which was previously found to be the most influential in a desktop environment. An important note is that the tests conducted in [5] assumed maximum Web site loading time was between 5 and 6.5 s, that is, that these results are not necessarily valid in the case of higher PLTs.

In addition, Baraković and Skorin-Kapov [133] have extended their previously reported multidimensional analysis given in [5] with new findings explicitly exposing the relationships between QoE IFs and QoE features. In other words, they have quantified mutual relations between four identified key influence factors (number of taps on a touch screen needed to achieve a given task, Web site loading time, Web site aesthetics, and quality of information) and the QoE features (perceived Web site loading time, perceived usability of Web site, perceived aesthetics of Web site, and perceived quality of Web site information). Their extensive continuing work contributes to the QoE research community in several ways. Namely, key features contributing to and describing QoE are addressed as multidimensional concepts. Then, as stated, mutual relations between QoE features and considered influence factors are quantified (multidimensional equation models), thereby extending and deepening the proposed multidimensional QoE models for mobile Web browsing given in Baraković and Skorin-Kapov [5]. Also, based on proposed multidimensional models, one is able to identify the importance of distinct factors in terms of considered perceptions and consequently overall user perceived QoE.

Lessons learned: summary of key findings

The tables provided within this paper summarise selected existing literature in the field of QoE modelling for Web and mobile Web browsing and relevant QoE and UX research according to the following parameters: addressed IFs according to the categorization given in the Qualinet White Paper [64] and the number of considered levels, considered features/dimensions, type of Web task, and consideration in a mobile context. Furthermore, the comparisons address the study specific parameters such as type of study that has been conducted, size of sample, used scale, conducted analysis, and manner of representing the resulting model. Finally, the tables contain the key findings of each study that have already been discussed in previous subsections. However, in order to summarise and highlight the main findings that have been reported in the surveyed papers, which may be of interest to both practitioners and researchers, we provide Table 5. In addition, we need to state that the findings and results of all addressed studies need to be validated in real-life settings and further discussed in order to make any final and general conclusions. Not having participated in the surveyed studies, it is often difficult to determine their ecological validity, that is, to what degree experimental findings mirror what can be observed in the real world.

Table 5 Summary of key findings
Full size table

Directions for future work: setting a research agenda

In previous sections, we have provided the key findings that may be of interest to both practitioners and researchers. Here, we provide a meta-analysis aimed at setting a research agenda in the field of studying QoE for Web and mobile Web browsing and conclusions that practitioners and researchers can find useful in their future work. Beginning with IFs, one may conclude that most literature considers only one or a limited set of factors exclusively originating from one group (user, system, or context), while the remaining factors are perceived as constraints in the testing methodology and are not taken into account. Most considered IFs fall into the category of system factors, while only several authors have examined the impact of user and context IFs in the field of QoE modelling or UX research for Web and mobile Web browsing, while even less took a multidimensional approach. Further on, most studies address composite factors (aesthetics, quality of information, etc.) and manipulate them in a low number of degrees, which limits the insight into their true impacts on QoE and user satisfaction. With regard to QoE features/dimensions, one may conclude that the most dominant ones are: perceived waiting time, perceived aesthetics, perceived usability, and perceived quality of information provided by the Web content. Namely, studies listed in Tables 1, 2, 3 and 4 and those described in previous sections have addressed various dimensions that may be subsumed under these four features/dimensions. By addressing a limited factor and dimension set, together with factors being manipulated in low number of degrees, the issue of QoE modelling is only partially covered, and may be considered as not being sufficient for a holistic understanding of Web browsing QoE. A limited understanding of QoE IFs further leads to limitations in terms of opportunities for managing and optimizing QoE.

Conclusion I

From our conducted survey of existing literature, the range of factors affecting Web browsing QoE and features/dimensions describing it is wide in scope: temporal, aesthetics, usability, screen size, brightness, previous experience, memory effect, task, movement, etc. Therefore, it is necessary for the research community to extend the scope of studies, that is, properly investigate (manipulate in more levels) the impact of a larger number of abovementioned factors (constructs or basic ones from all groups, that is, system, context, and human) on QoE in Web and mobile Web environments. Next, the key QoE IFs and features must be identified and mutually correlated, given the fact that addressing all of them for obtaining accurate QoE models is clearly not possible, especially in a simultaneous fashion. In other words, it has been recognized that there is the need for developing relevant and accurate multidimensional QoE models for desktop and mobile Web browsing based on key IFs and QoE features/dimensions. The multidimensional models would result in a better understanding of QoE and consequently its improved management.

Tested Web sites in the addressed studies range from informative and thematic ones to online photo albums and e- or m-commerce sites. Most researchers has conducted experiments in a single Web page context, but as stated previously, Web browsing does not issue a single request which is then answered by a short single media experience as with audio and video, but is rather realized as a series of requests and responses. Namely, when a single Web page load is considered, one may catch certain regularities in terms of observed parameters (for example, page loading time) and QoE and then create the model which would describe that relationship. On the contrary, in a real Web browsing context which may be considered as complex due to duration and exposure of users to the influence of various factors, it is hard to draw conclusions on described relations, while the ones drawn in simple scenarios hold up only to a certain extent. When it comes to usability, aesthetics, or information related to a Web site in single or complex Web browsing scenarios, their impacts on QoE still remain to be investigated.

Conclusion II

The evaluation methodologies for Web QoE must ensure that request-response patterns are issued throughout the test evaluation. In order to approximately address the real nature of Web browsing, tests must either define a fixed number of requests, as in studies of Hoßfeld et al. [44] and ITU-T Recommendation G.1031 [51], or define a fixed duration for one Web session that is being evaluated, as done by Reichl et al. [93], Schatz and Egger [103], or Schatz et al. [105]. The former approach enables researchers to exactly control waiting times as one of the main IFs, while the latter guarantees a more realistic web browsing experience for the user.

Conclusion III

There are unexplored and unrealized research issues in investigating and quantifying QoE for different types of Web content used by users such as that have not yet been addressed by discussed studies.

Further on, subjective user studies typically carried out by a test panel of real users in a laboratory environment are unique data collection methods of considered approaches. Several of them are being carried out in real world environments [103, 105], by using data sets collected by cellular networks [1], while some by using crowdsourcing methodology [121, 122]. Methods comprising laboratory settings have both advantages and disadvantages. The former comes in terms of a high level of control over test variables and the test environment, while the latter is represented with characteristics such as high costs and time-consumption. These issues become more important when the test design relies on iterative experiment conduction as it is usually applied when it comes to QoE modelling. In addition, this method has constraints in terms of sample size. On the other hand, conducting experiments in a real world settings may result in more accurate and valid findings, but these kinds of studies are difficult to control and still do not solve the problem of cost and time-consumption. Crowdsourcing enables the collection of a statistically relevant amount of data from a large number of internationally widespread users. This method also reduces costs and time for tests. This method has shown good results in several recent studies that investigated desktop-Web QoE such as Chen and Dhillon [16], Hoßfeld et al. [44], Keimel et al. [57], Varela et al. [122]. However, for the purpose of mobile Web QoE evaluation, subjective user assessment in laboratory settings is the most acceptable, since the other two (real environment and crowdsourcing), despite the possibility to collect larger amounts of data from the real environment with less cost and time, do not provide sufficient control over the mobile environment and context (for example, network performance, used device, or surrounding) that may have a strong impact on users’ ratings.

Conclusion IV

Since there is no answer to the question of which is the most appropriate approach to investigate QoE, it has been argued that the important question is not if or why one should do lab, field, or crowdsourcing studies, but rather when one should do what and how one should then do it. A promising approach may be to conduct complementary tests in both lab and field settings so as to compare results and draw clearer conclusions with regards to the impacts of “real-world” settings. Generally, it is important to achieve a positive answer to the question of study results being generalizable across different settings (ecological validity). In other words, one must have comparable results regardless of the environment in which the data has been collected to state the validity.

Moving from methods for collecting ratings to their analysis, based on the surveys in the previous tables, one can conclude that addressed studies used a whole range of statistical analysis methods including descriptive statistics, correlations, as well as regressions. In addition, those tools were applied on ratings that were collected based on the usage of MOS, 5-, 7-, and 10-point Likert scales. Also, as mentioned and discussed several times above, studies that deal with QoE or UX research in the domain of Web browsing are mostly descriptive or present their findings regarding the relations among considered factors, features, and user satisfaction or QoE, that is, models, by using graphics, tables, and statistics. Only several have provided concrete equations that may be used for estimation of QoE (for example, [5, 28, 46, 51, 70, 83, 93]). This represents a serious issue in subsequent QoE management and optimization processes, which are inherently dependent on QoE modelling.

Conclusion V

An important consideration is the notion of how to utilize the obtained understanding of the QoE of Web sites. Considering this from a practitioner’s point of view is crucial in order to set the grounds for improving or optimizing user experiences. Consequently, there is a need for more quantified descriptions of the impact of various considered IFs on QoE and its features/dimensions, given the fact that most addressed studies are qualitative. Research results should provide clear take-away findings for practitioners such as Web site content developers, device developers, and network operators. Thus, understanding the key factors that need to be considered and optimized in given contexts is of crucial importance.

However, as it is the case with all studies, this review has its constraints which reflect in a limited number of studies addressing and modelling QoE in the Web and mobile Web context given that this topic is quite young. As stated previously, we needed to involve studies dealing with qualitative descriptions of user experience and QoE in this field, which, even though the authors searched for all possible research articles that have results that could be linked to this review, may have resulted in relevant articles being omitted in this process.

In summary, it is the hope that the above findings and conclusions, as well as the outlined research agenda will stimulate further research in this research domain, and help researchers to improve the understanding, modelling, and optimization of QoE in the context of Web and mobile Web browsing.