1 Introduction

The Internet of Things (IoT) combines billions of devices (things) that can communicate via the Internet (e.g., wearables, smart-phones, sensors, industrial machines, RFIDs, etc.) and can sense the physical world, send observation data and actuate. In 2014, IoT covered 13% of the business, which rose to 25% by 2019. According to a report published in November 2019 by Statista Research Development, the total number of connected IoT devices is projected to be 75.44 billion worldwide by 2025, which is a five-fold increase in five years. According to McKinsey’s report on the global economic impact of IoT, in 2025, the annual economic impact of IoT will be in the range of $2.7 to $6.2 trillion [1]. IoT is growing rapidly and is finding substantial applications in industries like transportation, healthcare, manufacturing and retail. In these industries IoT assists in fine-grained process monitoring, generation of valuable insights and recommendations for decision making, and even putting decisions into action through actuation. For instance, the manufacturing domain has seen a significant influence from IoT, as it underpins the Industry 4.0 vision, that focusses on data-driven practices in manufacturing [2]. Industry 4.0 is used to improve the efficiency of plant operations through reduction of production downtimes and improved product consistency, thereby increasing the manufacturer's ability to be agile and meet the fluctuating customer demands [3, 4].

Let us consider an example of Industry 4.0 use case. Food products such as jams, food pastes etc. require processing to ensure that they meet the target product quality attributes, an important component of which is the consistency [5]. The food processing involves the use of a machine that performs boiling or evaporation to achieve the targeted paste consistency, which in itself is determined based on the key performance indicators (KPIs) of the manufacturing plant and product requirements. An IoT application [6] in this scenario can help achieve the target paste consistency by automatically (1) collecting data from the machine sensors and other sensors such as refractometers, (2) analysing this data to extract high level knowledge and (3) using this knowledge to provide actuation (machine settings) to the evaporator/boiler. The QoE of such an IoT application would be based on whether it leads to achieving the targeted paste consistency. Figure 1 shows the lifecycle of such an Industry 4.0 IoT application to highlight the challenges and need to measure QoE in autonomic IoT applications that involves device to device communication, analytics to extract insights/knowledge and actuation to perform an action.

Fig. 1
figure 1

Industry 4.0 application for ensuring consistency of product quality

Full size image

When IoT applications are developed and deployed to support such an Industry 4.0 application use cases, it is often difficult for an application designer to assess the quality the application delivers (e.g., ability to meet paste consistency as specified via KPIs and product quality specifications) due to multiple reasons. Firstly, there is typically no human feedback available based on which the QoE of the application can be estimated in real-time, since the manufacturing KPIs are typically not measured during production. Further, since the result produced by the IoT application i.e., paste consistency, is not just from actuation (i.e., applying specific settings to the evaporator machine), but is a result of other components of IoT, which includes the sensor data collected in real-time as well as the data analytics performed using the collected sensor data, evaluation of how individual components impact the final application quality is a grand challenge. The food processing use case outlined is a good example of such an autonomic IoT application that involves automated data analytics-based actuation.

As discussed in the above use case of Industry 4.0 food processing application, getting a desired consistency is not only an outcome of machine actuation but it involves other components of an IoT application such as data sensed at sensing stage. Further, the sensed data is collected and sent to distant cloud or nearby edge device for analytics. Based on the analysed information, machine settings are adjusted to obtain the desired consistency. The desired consistency is defined through KPIs. Such IoT applications involves sensing, analytics and actuation. This distributed and autonomic nature of IoT applications make it challenging to effectively measure the quality of their outcomes against the defined KPIs. It arises both due to the lifecycle of IoT applications (sensing, analytics and actuation) as well as their distributed architecture because humans cannot always be present to provide feedback at different stages of an IoT application lifecycle which are, sensing, analytics and actuation.

In data analytics stage, IoT application derives actionable insights from the data. Data analytics is the science of analysing raw data to extract valuable knowledge essential to decision making [7]. In IoT applications, IoT devices such as machines or sensors perform data analytics to extract high value information from data sensed by these IoT devices. The extracted knowledge is fundamental to drive actuations in IoT. Actuation is the ability of an IoT device to operate and make changes to the physical environment, as opposed to sensing which deals with measuring the physical environment. For example, in food processing application, machines interact with each other and collect data from machine sensors. This data is analysed to predict machine settings which can lead to desired consistency of the food product. Such advances in techniques for conducting predictive and prescriptive data analytics and increased availability of data via the IoT devices is shifting IoT to analytics-driven actuation i.e. IoT application are increasingly being developed to extract high value knowledge from the IoT data that allows them to take actions on behalf of humans [8].

There are several stakeholders in an IoT ecosystem (which we describe as a system of systems). These include cloud and network providers, application developers, the end-users, as well as the IoT devices (e.g. machines in case of Industry 4.0) themselves [9]. For example, the food processing application (discussed above) involves machine to machine interaction at sensing stage, data analytics on collected data in cloud or edge. The data collected by machine sensors is sent to distant cloud via network or a nearby edge device and finally actuation done as automatically adjusting the machine settings to obtain a desired consistency of the food paste. Such applications are called autonomic IoT applications. Similarly, in [10] authors proposed an early fire detection system to prevent fire events. The proposed fire detection system is a combination of multiple technologies such as wireless sensor networks (WSNs), unmanned aerial vehicles (UAVs), and cloud computing, as well as image processing-based data analytics. The components provide several services to each other for operations within the IoT ecosystem (a classic example of systems of system). For example, the Platform-as-a-Service (PaaS) cloud provider provides application developers application hosting services, and the network provides the network connectivity services. Furthermore, to support the increasing need for real-time processing in IoT, edge computing model aims to provide processing and storage capacity at the network edge. This enables analytics to be carried out at the edge without the need to move all the data/processing to a distant cloud datacentre [11]. However, the introduction of edge computing introduces additional complexities due to its highly distributed nature and need to support distributed processing of IoT application. This distributed processing of IoT applications further necessitates continuous evaluation of quality of IoT applications. Till-date the notion of network-, cloud-edge and IoT-based application performance hinges upon QoS assurances that are typically guaranteed via service level agreements (SLAs) [12]. QoS metrics such as throughput, delay, jitter, packet loss, availability and reliability are precise in definition but are rather crude. For example in [13], multi-mobile sinks-based QoS-aware data gathering protocol (called MQRP) for WSNs-based smart grid applications has been proposed to empower smart grids. The experimental results in the paper show that the proposed scheme leads to improved QoS performance metrics, such as packet delivery ratio, memory utilization, control message overhead, residual energy, network lifetime, and throughput. While these metrics indicate improved network performance, they do not provide an overall view of the quality of the IoT application (e.g., from data to actuation). However, in IoT applications, quality is not driven by network metrics alone but also by other metrics such as data. In surveys such as [14, 15], authors have focussed on QoE in multimedia IoT applications, which is determined based on how such applications are consumed by users. However, the outputs of an autonomic IoT application may be consumed by another IoT device (e.g. machines in case of Industry 4.0), implying the absence of any user feedback.

Existing surveys on QoE in IoT such as [16,17,18] focus on network metrics to assess QoE. Recently Minovski et al. [19, 20] and Mitra et al. [12] have argued for the need to understand quality in an IoT ecosystem; within an IoT ecosystem a user can be both human and/or other systems and IoT devices. Measuring QoE provided by an application is an important metric for system designers and developers as it provides them with a mechanism to quantify the outcomes provided by the application they are developing. Typically, QoE is measured using subjective tests where the users (or subjects) rank the quality of service or applications perceived by them using Likert-like scales and based on the provided ratings, some conclusions are drawn. For example, when a user watches a video via the internet, he/she may give subjective feedback of his/her experience of the application, which may be a result of multiple factors such as the video resolution and buffering. Existing literature on QoE focuses mostly on subjective human feedback of applications [21]. However, as mentioned above, in an IoT ecosystem, there is a need to consider device to device interactions that can impact the quality of the application and hence the need for more objective measures to assess QoE [19, 22].

The need to assess QoE objectively is enforced more strongly with the rapid increase and adoption of analytics-driven actuations in autonomic IoT applications where there is no human feedback. While existing definitions of QoE cover the satisfaction of end-users, they may not be sufficient for IoT applications where an actuation performed by IoT device (e.g., machine) can impact another IoT device or a set of connected IoT devices (and not always a human). In summary, for applications where IoT device-to-device interaction and actuation happen without humans in the loop, existing QoE measurement methods such as subjective human feedback is not enough to measure the application quality. Furthermore, the distributed and heterogeneous nature of IoT applications implies that the quality of IoT applications is impacted by multiple factors, which make them distinct from other QoE measurement methods in the literature. This paper aims to provide a vision and identifies key challenges in evaluating quality of experience in Internet of Things applications without human involvement. Its main contributions are:

  • Review of existing QoE definitions and analysis of how they apply to autonomic IoT applications.

  • Review of existing literature on QoE in IoT, including the state of the art on measurement of QoE for IoT applications, QoE models in IoT and the role of data in evaluating the quality of IoT applications.

  • A discussion summarising gaps in existing research on QoE in IoT, and a vision identifying future research directions.

The rest of the paper is organized as follows. Section 2 presents an overview of the methodology we have employed to carry out the survey and presents existing QoE definitions from the literature. Section 3 presents a review of existing literature on QoE in IoT applications. In Sect. 4 we discuss the research gaps and future research directions. Finally, Sect. 5 concludes the paper.

2 Survey methodology and QoE definitions

2.1 Survey methodology

In this subsection we describe the methodology used for carrying out the survey in this paper to answer the key research questions: “What are the current state of the art techniques and approaches in evaluating QoE of autonomic IoT applications”.

In this paper we have carried out a survey of the research works published from 2007 to the present (2020), referring to QoE in IoT. To carry out the survey, the scientific databases used were Google Scholar, Springer, Science Direct, IEEE Xplore. The databases used include the largest body of scientific information and allow finding and accessing articles in highly ranked conferences and journals, as well as in other repositories of scientific texts. For searching these databases and based on the research question the following key terms were used: “Quality of experience” and “IoT”, “QoE definition” and “IoT”, “QoE” and “data”, “QoE models”. These terms are searched in Abstract/Title/Keywords, from 2007 to the present. Figure 2 shows the search strategy used in this research; the criteria we have used are provided by the specific search engine of databases.

Fig. 2
figure 2

Search criteria in different databases

Full size image

For the selecting the papers, we first read the titles and abstracts of studies found. The selection criteria to consider classifying the article were the following: (1) Studies of QoE and IoT; (2) Studies of QoE definition and IoT; (3) QoE and data; and (4) QoE models. We have classified the papers by reading their abstracts as well as the full paper when necessary; all the papers repeated in more than one database are eliminated. By surveying the literature, we find a considerable number of studies that base their research on QoE and IoT. In total, we selected 50 papers, most of which are from past 10 years.

2.2 QoE definitions

In this subsection, we review the QoE definitions proposed by different standard organisations and research articles based on the paper we selected using the methodology presented in Sect. 2.1. We also discuss how these existing definitions are adapted for applications which have human in the loop to provide their feedback, or the experience received from the application but lack to cover IoT applications where human is out of the loop.

Organizations like International Telecommunication Union ITU-T (ITU-T) and The European Telecommunications Standard Institute (ETSI) have defined QoE. Such QoE definitions proposed by standard organizations are listed in Table 1. Various application domains such as video conferencing, mobile applications, multimedia IoT and IoT have adopted these QoE definitions. The definition given by ITU-T has subjective measure of an application or a service received by the end-user. In contrast to the definition from ITU-T, the definition of QoE from ETSI includes a subjective and objective measurement to complete the blueprint of QoE definition. Subjective human factors include usefulness, usability, need, availability, feelings, happiness, boredom, expectations, desires, social influences, brand image, satisfaction, etc.

Table 1 QoE Definitions
Full size table

On the contrary, objective human factors such as reaction time are quantitative. Subjective evaluation is considered as the most accurate method of getting a user's opinion. However, it is a complex method that fails for real-time applications [23].

To the best of our knowledge there is no widely generic definition of QoE in IoT. However, all the existing definitions of QoE focus on the understanding and measurement of the quality of an application or a service received by the end-users (humans), and typically from multimedia applications perspective. Here the QoE is evaluated based on the subjective feedback provided by humans. For example, in [30], authors have used the QoE definition given by ITU-T and they describe the IoT network as an aggregate of three factors i.e. system, content, and service. Content is the information that IoT carries. However, in their work, they do not directly measure the quality of content, service, and system. But they measure these metrics through the utility and hedonicity the end-user (humans) receives. Since IoT is a network of devices connected via the Internet, the application receiver is not always essentially a human [19].

Regarding IoT, the ITU-T definition for QoE is insufficient. Similarly, definitions of QoE given by the European Telecommunications Standard Institute and Qualinet White Paper cannot be mixed with IoT in its purest form because IoT application involves IoT device to device interaction and humans may not be present as end users to provide his/her feedback about the application. None of the existing QoE definitions aim to evaluate quality of IoT application in terms of sensing, analytics and actuation. Also, these definitions involve human feedback to evaluate the application quality and do not include machine experiences; hence., quality measurement of such IoT applications remains uncovered.

In the existing literature on IoT, we found only one reference which looks to define and evaluate the quality of IoT applications [19]. Here the authors coined and defined the term Quality of IoT-experience (QoIoT), which measures the performance of an IoT service from two different perspectives: (1) user-centric quality, termed QoE; and (2) machine focused quality, termed Quality of Machine Experience (QoME). QoE considers factors from the physical devices, network, and other application-specific metrics and maps it to the users' subjective assessment. QoE considers only the end-user’s objective and subjective experience but does not measure the overall machine performance. QoME measures machine performance objectively. Machine performance is measured in terms of decision-making, reasoning, and optimized machine operations.

3 Evaluating QoE in IoT

In this section, we present a review of existing literature on QoE in IoT. First, we survey existing methodologies and techniques for measuring QoE for IoT application. We then study the existing QoE models for IoT applications that have been proposed in the literature. Finally, we present an analysis on how role of data plays paramount significance in evaluating the QoE of IoT applications.

3.1 QoE models for IoT

In this section, we review existing literature in evaluating the quality of IoT applications where, the quality is evaluated subjectively or objectively. In subjective evaluation, users rate the quality of the application on a measuring scale such as the Likert scale with ratings 1 to 5 where ‘1’ means “poor”, ‘2’ means “fair”, ‘3’ means “good”, ‘4’ means “ very good”, and ‘5’ means “excellent”. After the subjective ratings are gathered, their average is taken and a score is drawn, for example, the mean opinion score (MOS) [31,32,33,34]. Even though using MOS is the most widely used method, its validity is recently questioned by several researchers [35]. Objective metrics evaluate the quality of applications through mathematical/statistical/probabilistic models.

One of the recently evolving methods for measuring the quality of multimedia content involves both subjective and objective evaluation to obtain the best of two methods [36]. For example, in one of such works [31] on QoE in multimedia IoT, authors measure QoE objectively. First, they define multimedia IoT application as a network of interconnected objects capable of acquiring multimedia contents from the real world and/or present information in a multimedia way. They use smart surveillance as an example application where smart cameras record images and audio of a place. This multimedia information is collected by an IoT application in the form of multimedia data, which can be images, videos, and audio to provide a remote security control service. Another example used by the authors in this paper is of a restricted venue that provides entry to only authorized people. A camera records images of people who want to enter the place. The recorded images are sent to an IoT application that consists of an identification software that determines whether the person recorded by the camera is authorized to enter the place. If the software identifies the person, a command is sent to the door actuator to open the door, else the door remains closed, and access is denied. The quality of such multimedia IoT applications can be evaluated in terms of how accurate the actuation command is. If it is sent only for authorized people, the quality is good else if it also allows non-authorized people, the application is of poor quality.

In [40], the authors categorized the metrics which impact application QoE into two broad categories: physical metrics and metaphysical metrics. Physical metrics include the factors that impact the application quality and are mapped to IoT architecture. Metaphysical metrics correspond to factors which impact the application quality and are mapped to application requirements. To organize the quality metrics corresponding to IoT architecture (physical metrics), functions of IoT architecture are virtually divided into four different layers and the quality metrics are defined for each layer. These layers are device, network, computing, and user interface. In the device layer, metrics related to data producing IoT devices are considered such as, GPS accuracy of sensor devices or the resolution of camera devices. The time duration for which sensor is sensing also comes under the device layer. However, the authors do not highlight how these metrics impact the quality of IoT applications. The same is true with some of the other existing literature [30, 37, 38], as they do not assess how the application QoE relates to the individual components of IoT applications. They model the application QoE and measure the users experience through subjective evaluations. One such work is [42], in which authors measure the QoE of teleconsultation application called Psiconnect using an approach that is based on pentagram model. This pentagram model is based on five factors (integrality, retainability, availability, usability, and instantaneousness). Using this model, they compute QoE value. The results obtained from this quantitative model gives a value which infers how much the users are satisfied with the use of a teleconsultation application.

In [39], the authors proposed a QoE model based on a generic QoE equation. They modeled QoE as a weighted linear combination of Quality of Data (QoD) and Quality of Information (QoI). They assume that the measured QoE value from the model lies between 0 to 5. They split the data into two equal halves. One set of data is used to build the model and the other is used to test the accuracy. They evaluated the accuracy of their model by conducting subjective tests. For each user, there are two results, one from the subjective test and other from the model. The results from the proposed model shows a strong correlation with the results obtained from subjective test through a Pearson correlation value of 0.81.

In [43], the authors measure the QoE of a smart city application using the smart city of Jakarta as a use case. In this use case, authors first identified the quality requirements of a smart city. Secondly, they defined the users of this IoT application. Thirdly, they compute a mean opinion score (MOS) based on surveys performed before and after implementing the IoT application. Lastly, the difference between after implementing and before implementing the application is performed. Based on the results obtained, they have built a relevant strategy to improve the quality of the application such that the user's experience is enhanced.

In [36] authors developed a predictive model for quality evaluation in IoT. The model is based on user experience from the IoT application. This model is solely based on data gathered from the survey performed on the users of the application. Layer based model is another approach for modelling quality of IoT applications. Realizing the fact that layered approaches are widely used in network models for their functionality, such as the TCP/IP and OSI networking models, authors extended the layered base modelling approach in the IoT domain. In [40], authors aim to ensure QoE through existing QoS parameters. In this direction, they develop a functional model between QoE and QoS. The model uses principal component analysis (PCA) to identify the principal components. Next, they deduce regression equation between QoE and principal components. Based on this regression equation, they obtain a functional model which connects QoE and QoS.

In [41], authors proposed a QoE model to measure QoE of a teleconsultation app. First, they identified five factors which impact QoE. These factors are integrality, retainability, availability, usability and instantaneousness. These five factors are measured using mathematical equations and the value lies between 0 and 1. Finally, QoE value is computed using the values of these five factors.

In [42], authors proposed a layered model for multimedia IoT applications (MIoT). The proposed model is composed of five layers: Real-World Layer, Network Layer, Virtualization Layer, Combination layer, and Application Layer. In real-world layer, physical multimedia IoT devices collect data and send it to the virtual layer through the network layer. The network layer accounts for traditional QoS parameters such as delay, packet loss, and jitter which can measure the performance of data transfer. In virtual objects, functionalities of the real-world objects like IoT devices are virtualized through virtual objects. In the combination layer, composite virtual objects are created. It is composed of different virtual objects to provide determined service which cannot be provided through a single virtual object. In the application layer, QoE metrics focused are control, interactivity, and presentation. This proposed layered QoE model aimed at combining the contributions of QoE metrics from each layer to measure the overall QoE in IoT applications.

Table 2 summarizes the QoE models in IoT, metrics and the dataset used for the model evaluation. The QoE models we reviewed in this section can be classified broadly into two categories as follows:

  • Methods in which the application QoE is broken down into sub metrics such as quality of data (QoD), quality of information (QoI), quality of service/network (QoS) and application QoE is measured in terms of these sub metrics.

  • Methods which evaluates the application QoE without breaking it down to sub metrics and used subjective methods based on users’ feedback. These models are based on architectural layers of IoT. This method has a major drawback because there is a lack of general IoT architecture in literature.

Table 2 QoE models, dataset and metrics used
Full size table

In summary, while reviewing the current literature, we identified that QoE is generally measured based on human feedback or the experience they receive from the application (such as in multimedia IoT applications). In most cases, the QoE models (described above) computed QoE either using synthetic data or data from human feedbacks. However, for autonomic IoT applications such as the food processing Industry 4.0 application (discussed in introduction), human feedback may not be always available. Hence, for such applications, we need to identify the metrics or factors on which the application quality depends.

3.2 Role of data in QoE of IoT applications

The existing literature in evaluating quality of IoT applications are dependent on the data which application senses, analyses and use it to actuate. This makes data an important metric which can impact the quality of the application. Given this, we surveyed existing literature to identify the factors on which the quality of data depends.

The authors in [45] identified that quality of data is crucial not only at device or network level but also at application level. The paper identifies challenges and propose solutions of impact of data on different Open Systems Interconnection (OSI) network model. They have used IoT- enabled digital health applications to measure the data quality. However, their approach involves human feedback to evaluate the application quality.

In [46], authors study the data from smart city application to evaluate performance of data driven QoS/QoE framework. Similarly, in [43] authors used the concepts of quality of data (QoD) and quality of information (QoI) to map a relationship between human experience and quality perception. For their work, they used a smart wearable application in which QoD is concerned with the accuracy and precision of the data collected by the smart-wearables while QoI relates to the useful information that is obtained from the unanalysed data collected through the application of each smart wearable installed on a smartphone.

In [47], authors identified that a poor quality data likely leads to unsound decisions and actions in IoT. The paper aims at improving data quality by identifying data properties, data lifecycle in IoT and the factors which can impact data quality in IoT. They also identify data outlier as one of the factors, which have an adverse impact on the data quality. Likewise, in [48] authors draw a mapping between data quality and network metrics to analyse the impact of data quality on network. Minovski et al. in [19] evaluated the quality of autonomous vehicles based on metrics such as quality of data, network and context. They identified the importance of accurate sensor data for collision free working of autonomous vehicles. The authors in [37] identified the role of data on application quality at various stages from data collection, data transfer, data analysis to user interface. In contrast, the authors in [49] conducted an in depth analysis of impact of quality of data and information in IoT. In [30], authors evaluate the quality of experience in IoT based on quality of network which in turn is driven by the generated IoT data.

In summary, an analysis of the existing literature on QoE in IoT leads to a conclusion that data plays a crucial role in quality of IoT applications. From our perspective, data quality can be characterized by the metrics: accuracy, completeness, usefulness and timeliness. Accuracy refers to the precision of the collected data. Data completeness is the ratio of the amount of data collected to the total amount of data required. Usefulness refers to how useful is the sensed data for the application. Timeliness is, how valid is the collected data for decision making. Table 3 lists such existing papers which has characterized IoT data based on the metrics such as accuracy, completeness, usefulness and timeliness.

Table 3 Characterizing IoT data
Full size table

4 Research gaps and future research directions

QoE in IoT showed is a well explored area. However, QoE evaluation in existing literature for IoT application is either subjective or objective. Subjective feedback involves human in the loop who provide their feedback based on which application quality is evaluated while objective feedback has no human in the loop. The plethora of the paper we have surveyed (53 papers) have considered human involvement in QoE evaluation for IoT as shown in Fig. 3. X-axis of the bar graph is labelled with the percentage value and Y-axis is labelled with the type of feedback involved in QoE evaluation. We found that 77.77% of the work in the literature (blue bar) involves human with their subjective evaluation for QoE in IoT. Subjective evaluation means that human provide their qualitative feedback in terms of metrics such as Mean Opinion Score (MOS), grading scale, etc. while 22.22% of the work (grey) involves no human feedback for evaluation.

Fig. 3
figure 3

Human Involvement in QoE Evaluation in IoT

Full size image

Based on the above gaps identified in the literature, we present our vision and future research challenges that needs to be solved for evaluating QoE of autonomic IoT applications.

4.1 QoE definition specific to autonomic IoT

QoE in IoT needs to be redefined to comprehensively incorporate the characteristics of IoT applications. Key to our vision of a definition of QoE in IoT is the fact that application quality need not always be measured based on user feedback, even though Key Performance Indicators (KPIs) may be derived based on end user or business requirements. For example, consider a smart parking application for an autonomous car which is an IoT device receives parking recommendation [51]. This is because IoT applications may operate automatically without human intervention and even produce outcomes that are consumed later. Further, a comprehensive definition of QoE in IoT also needs to account for the fact that IoT applications are often systems of systems, in which multiple autonomous IoT applications interact with each other. This implies that the outcomes of an IoT application may be consumed by another device or even another IoT application. To the best of our knowledge, Minovski et al. [11] provided the first comprehensive definition of QoE from IoT perspective. It would be interesting for industry and academia to see the development of additional definitions, especially from standardization bodies such as ITU and ETSI.

4.2 Factors impacting QoE of IoT applications

A primary purpose of QoE is to enable an application designer to evaluate the performance of the application and how individual application components contribute to the performance. From this perspective, we envision research on QoE in IoT to determine how various factors contribute to the QoE of the IoT application. IoT applications are distinct from other applications in the fact that not only are they inherently distributed, but the distribution is result of both computation requirements and the distinct roles of application components that contribute differently to the application outcome. Thus, it is necessary to determine how quality factors such as data, network, analytics, etc. impact different stages of an IoT application lifecycle i.e. sensing, networking, data analytics and actuation impact the application quality.

4.3 Measuring QoE for fog/edge enabled IoT applications

In addition to the factors mentioned above, the distributed architecture of an IoT application also significantly impacts its performance [52]. The distribution of application processing across edge and cloud, as well as the networking among them, impacts the responsiveness of the IoT application. IoT applications' distributed processing on the network edge and the centralized cloud is suitable, especially for latency sensitive IoT applications. Fog/edge enabled applications may work in real-time. Thus, first, we cannot use the subjective tests in such cases, and objective metrics are necessitated. As discussed till now, the notion of QoE in IoT is relatively new; therefore, new objective metrics need development to be used in IoT contexts. Second, novel methods for real-time QoE estimation will be required using either active or passive probing methods. For instance, in the case of active probing, additional data requests are generated to mimic realistic IoT traffic workload. This workload characterization can be used to build QoE estimation models. In the context of passive probing, the traffic generated either at the application layer or the network layer can be observed by the entities in the IoT ecosystem to estimate the QoE indirectly. For example, authors in [48] used artificial learning-based models to estimate QoE in IoT, albeit for video traffic.

4.4 Practical QoE models with no human involvement

Finally, our vision includes the development of a generic model of QoE in IoT which not only incorporates the various factors impacting the application performance but provides a way to map the impact of individual factors with the final application quality. Such a model enables the estimation of IoT applications quality based on the performance of individual factors such as sensor reliability impacting the application quality [53]. In Sect. 1, we have used an example of Industry 4.0 to exemplify the requirements of QoE models, which can measure QoE without any human involvement. Currently, QoE models are based on human feedback or experience. There are metrics that directly impact application quality. Suppose the designer of the application can identify the factors and evaluate their impact on the application quality. In that case, such QoE models can be used for applications where a human is not present to provide his subjective feedback. Since IoT applications involve a set of connected devices, a practical and objective QoE model is required to assess the quality of the application based on the data sensed, analysed and decisions taken by these devices.

4.5 Implication of security mechanisms on QoE of autonomic IoT application

Autonomic IoT applications are distributed over multiple networks owing to the distributed nature of IoT. The sensing component of the application in sensing stage of the lifecycle can be on one network while the data analysis component can be on some other network during analytics stage. This necessitates data transfer between different networks rendering threat to the security aspects of the applications [52]. To deal with security mechanisms, quality of application is compromise. For example, in food paste processing application, the IoT network is outside the corporate machine network. Hence, there are security issues that render delays in sensed data from machine being delivered for analysis.

5 Conclusion

A thorough understanding is required about how current QoE evaluation techniques lack to model and measure QoE of autonomic IoT applications where human feedback is not always available. This paper presents a vision for understanding QoE in IoT where the IoT application is comprised of system of systems, each interacting with one another to exchange data, extract knowledge and perform actuations with limited or no human involvement. This vision of understanding QoE not only ensures quality for autonomic IoT applications but also looks at QoE in IoT from the perspective of the designer of an IoT application. While the literature has covered and provided a significant understanding of QoE via collection of subjective feedback obtained from humans, evaluating QoE from the notion of IoT systems (devices, machines etc.) as users is limited. We reviewed the existing literature including QoE definitions, all of which are focused on human as users and QoE evaluation is based on their feedback. We conducted an in-depth analysis of QoE models, metrics used to measure QoE in IoT and identified that data plays a crucial role in quality of IoT applications. We identify that there is still a gap in QoE evaluation for IoT applications and researchers can contribute to the area of QoE evaluation-based on objective feedback. Finally, we provided future research directions that focus on the need to identify quality metrics at different stages of the IoT application lifecycle which can impact quality of IoT applications. There is a need to develop QoE models based on objective feedback for quality evaluation of autonomic IoT applications.