Keywords

1 Introduction

The Internet of Things (IoT) paradigm has recently emerged, and one of the areas that are leading up to IoT is Ubiquitous Computing (UbiComp) [32, 52]. The evolution happened due to the association with other areas, such as Cloud Computing, and the hardware advance, which allowed the improvement of the sensors, actuators and the creation of smaller devices with a network connection [1]. In this paradigm, we have several things communicating between themselves. A thing is an object/device of our daily routine with sensors, actuators, embedded computing and wirelessly interoperable [36].

Ubiquitous Computing (UbiComp), in turn, was defined in 1991 by Mark Weiser who considers that “the most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it” [55]. UbiComp makes then the technology present in our daily life able to monitor us and provide services, but we do not perceive that we interact with these technologies.

Although UbiComp has preceded IoT and we could identify commonalities between them, the challenges related to building IoT systems are far more complex. These problems arise mainly because of the number of things that make the IoT systems and the different role they play [18, 35].

Nevertheless, the interaction evaluation can be even more complex in IoT than in UbiComp systems once we have two different perspectives to take into account: Human-Thing and Thing-Thing interactions [30]. The first one is the interaction between users and things and has more commonalities with the UbiComp interaction evaluation [31]. The second one is the interaction between things themselves [51], which is both a novelty and a challenge.

In this paper, we investigate the innovations and issues introduced by things, considering the literature, our previous experience with Ubicomp [25, 26, 39], and case studies. Then, we discuss how we can benefit from the UbiComp move towards IoT, highlighting the main commonalities and differences related to interaction evaluation.

For the case studies on interaction evaluation, we choose three ubiquitous applications and two IoT applications. This evaluation was performed using a set of measures that focused on characteristics for UbiComp, such as context-awareness, mobility, transparency, attention and calmness [8], which are also presented in IoT Systems. The results showed that is possible to apply the same measures in both Ubicomp and IoT applications to evaluate user interaction. However, there are specific restrictions of IoT systems that also need to be considered and prioritized in the evaluation. So, we believe and discuss throughout this paper that Thing-Thing interaction severely impacts on Human-Thing interaction evaluation, for example, the responsiveness of the system changes because things are on the Internet [41] and the answer can take a while.

The remainder of this paper is organized as follows. Section 2 presents the main concepts about Human-Computer Interaction (HCI) Evaluation in UbiComp and a brief overview about IoT. Section 3 presents the case studies with UbiComp and IoT applications. Section 4 presents a discussion about open questions that allow the identification of important research challenges and a brief conclusion is presented in Sect. 5.

2 Background

In this section, we bring a brief overview of the ubiquitous computing concepts highlighting the context-awareness and other characteristics that impact the quality of user interaction in these systems. We also introduce the main concepts and elements of IoT to give a general outline to the reader.

2.1 HCI Evaluation in UbiComp

As we introduced before, the Ubiquitous Computing was coined by Mark Weiser in 1991 and became a well-established research area [55]. The main motivation behind UbiComp is to support the user with daily activities without the user noticing they are interacting with a computing infrastructure. Therefore, the technology should be invisible or cause a minimal distraction to the user, changing completely the way users interact with systems [45].

To achieve Mark Weiser’s vision of UbiComp, the system should be capable of understanding user’s behavior. This is enabled by context-awareness, which captures relevant information during the interaction between users and applications, and applies it to support users in performing their tasks [11]. Therefore, UbiComp allows that the systems capture various information (e.g., GPS, temperature, accelerometer, and magnetometer) without the user noticing, or by voice, gesture and touch. These new sources of inputs make the interaction with users much more natural [8] and allow the interaction being initiated by the system itself.

Regarding HCI evaluation in Ubicomp, a systematic mapping study that we performed in [8] identifies a set of 27 essential quality characteristics to evaluate ubiquitous systems that run in mobile devices (see Fig. 1).

Fig. 1.
figure 1

Characteristics for HCI evaluation in ubiquitous systems.

Full size image

All characteristics found in the systematic mapping impact, somehow, on the quality of interaction in ubiquitous systems and, therefore, need to be taken into account. However, there are general characteristics for any system (for instance, efficiency and security) and specific characteristics for ubicomp applications (marked in bold in Fig. 1). These particular characteristics are better explained as follows:

  • Context-awareness is the system’s capability of monitoring contextual information regarding the user, the system, and the environment. With this information, the system dynamically and proactively adapts its functions accordingly [20].

  • Mobility in Ubiquitous Computing refers to the continuous or uninterrupted use of the systems while the user moves across several devices [56]Footnote 1.

  • Transparency is the system’s ability to hide its computing infrastructure in the environment, so the user does not realize that is interacting with a set of computational devices [48].

  • Attention in ubiquitous environment refers to the system’s ability to keep the user’s focus on real-world interactions rather than in technology [17].

  • Calmness is the system’s capability of interacting with the user at the right time and situation, presenting only relevant information, using the periphery and the center of attention only when necessary and being easy and natural to use [38].

As these five characteristics directly impact the quality of interaction in UbiComp applications, they need to be taken into account by an HCI evaluation. One way of evaluating these characteristics is using software measures. Our systematic mapping in [8] found 218 citations of measures associated with these five quality characteristics. Most measures found are defined to evaluate the Context-awareness characteristic (44 measures)[8].

However, most of them are simple definitions with no detail about their measurement functions. Because of that, our group has been working on the definition of software measures for these characteristics [14, 15, 45]. For instance, Table 1 shows a set of software measures for the Calmness characteristic already presented in [6].

Knowing that IoT presents similar characteristics from UbiComp, for example, context-awareness [35, 52] and transparency [54], it is feasible to use software measures from this area to evaluate the interactions in IoT, i.e., Human-Thing and Thing-Thing interactions.

Table 1. Software measures for the calmness characteristic. Adapted from [6]
Full size table

2.2 IoT

According to IEEE [32], Internet of Things is a network of objects, which are embedded devices, connected to the Internet [32]. Considering this, Fuqaha et al. [1] define IoT to be formed by six elementary building blocks (see Fig. 2): (i) identification; (ii) sensing; (iii) communication; (iv) computation; (v) services; and (vi) semantic.

Fig. 2.
figure 2

The IoT elements. Source [1]

Full size image

The first element, the identification, allows an object to be accessed over the Internet by other objects, in a Thing-Thing interaction, or by an application controlled by the user, in a Human-Thing interaction. Here, the identification is comprised of two elements: (i) the network address of the thing, and (ii) the description of its functionality (e.g., light or A/C).

The sensing is responsible for capturing the information from the environment or for actuating to change that information. Examples of information that can be collected or controlled are luminosity level and temperature. Finally, information gathered from the environment or changed when the actuation is completed must be available on the Internet to be accessed by others things or applications [1].

The communication element is responsible for connecting a thing using the identification element. In IoT, there are several protocols to facilitate the communication, both at the network layer, such as Bluetooth, Wi-Fi, LTE, or NFC, and at the application layer, such as the Constrained Application Protocol (CoAP)Footnote 2 and the Message Queue Telemetry Transport (MQTT)Footnote 3. In the network layer, the characteristics of the communication technologies, like range, energy consumption, reliability or routing protocols, define their applicability according to the characteristics of the things. In the application layer, the functionalities required by the application guide the selection of the most appropriate protocol.

The goal of the computation is to process the information perceived from the environment, and to decide when acting on the environment is required, based on the processed information. The processing is executed both on the things embedded in the environment and on processing units located in the cloud. The decision to process it locally on the thing or on the cloud depends on the processing capabilities of the embedded devices, the load inserted by the processing algorithm and the response time required by the application.

The services are an abstraction used to represent the things, with two main responsibilities: (i) encapsulate the functionality provided by each thing; and (ii) provide a known interface to access that functionality. Services improve the interoperability; they hide details about the things, such as operating system or programming language [1].

And the last element, semantics, allows the knowledge representation of both the information collected by the thing and the information processed either by the thing or by a cloud unit [1].

These elements are present in different domains, like e-health, transportation, agriculture. For example, in healthcare, things (smartwatch, camera, presence sensor, heart rate sensor, among others) monitor an elderly in his/her house and the application sends the collected information to the physician, where he/she can control this information and improve user care. In transportation, for example, users can monitor the status of vehicles and they can send a command to change the behavior if it is necessary. Or yet, to control the growth of the plants in agriculture monitoring the sun rays, humidity, rain, among others.

In all these scenarios, the Thing-Thing interaction is present. The work from [51] calls the Thing-Thing communication as machine to machine (M2M). However, M2M refers to those solutions that allow communication between devices of the same type and a specific application, all via wired or wireless communication networks and does not generally allow for the broad sharing of data or connection of the things directly to the Internet [18]. In contrast, IoT also refers to the connection of such systems and sensors to the broader Internet. Therefore, the Thing-Thing interaction refers to the communication between things over the Internet [18]. Then, we believe that the evaluation for IoT should also take into account characteristics of the Thing-Thing interaction that is a critical part of the IoT environment.

Analyzing the IoT elements and the scenarios they are used, two UbiComp features can be identified in IoT as well: context-awareness and adaptability. Context-awareness captures the information from the user environment to provide a better service to the user. The Adaptability uses context information to make changes in the application behavior. For example, when the mobile application identifies the user location to be nearby his/her house, the application sends this information to the user’s home and the air conditioner is turned onFootnote 4.

In [36], another important feature influencing the application behavior is presented: spontaneous interaction. Razzaque et al. said that subtle interactions exist when users and things move around, coming in and out of their communication range leading to the spontaneous generation of events. Hence, subtle interactions between things have significant implications in the Human-Thing interaction as well. This characteristic can be compared with the service discovery and the context-aware from UbiComp. This occurs for the sake of the new services that can be provided and adapted based on the new devices in or out of the environment.

3 Applications and Evaluations

The applications that are presented in this paper are an essential part of the research conducted by the Group on Computer Networks, Software Engineering and Systems (GREat) of the Federal University of Ceará (UFC). They are a result of both research projects and undergraduate and graduate students work that were developed over the past ten years.

In this section, first, we present three ubiquitous applications as follows. GREatTour is a product of a Mobile Guide dynamic software product line (DSPL) [28, 29] and we have been using this application to evaluate several masters and Ph.D. theses in DSPL [2, 27, 42,43,44, 47] and HCI quality evaluation [16, 40, 46]. Both GREatMute and GREatPrint were developed in two different master theses to evaluate their findings [34, 46].

Then, we show two IoT applications, which are our first efforts to move from Ubicomp to IoT: GREat Room [10] and Automa GREat. Both applications are a result of a graduate course about IoT and they have been used as case studies for our research in IoT [4].

3.1 UbiComp Applications

GREat Tour is a UbiComp application to guide visitors in the GREat laboratory [29]. For each room visited, the application shows texts, images, videos, audios and persons who work there. GREat Tour shows this information according to the visitor’s location inside the lab. The application also presents some information according to battery level. For example, if the battery’s charge is more than 50%, the application presents all kinds of medias. If the charge is less than 50 and more than 30%, the application blocks the videos to the user. If it is less than 30%, the application does not show images and videos.

GREatMute aims to turn off the device’s sound according to the user’s appointment in his/her agenda and current location [6]. For example, if the user has an appointment 4 p.m. in an office registered in the calendar, the application will check if the user local is equal to the defined in the agenda in the hour recorded. If the answer is affirmative, the sound is turned off. Moreover, this application allows users to specify for which events they would like the device to be put into silent mode; this is done by monitoring for matches with the keywords registered in the system. When the application finds an event in which the title has one of the recorded keywords, GREatMute extracts the information of the start time and end time of the event, using this information to put the phone into the silent profile during this period.

GREatPrint aims to print documents at the nearest printer from the user, providing a ubiquitous and mobile printing service [7]. The application works as follows: after selecting a file, the user click on the print button of the given document, and the application collects the six Wi-Fi networks previously mapped in the laboratory. Then the application sends the document to the server. Based on the GREatPrint Server response, GREatPrint App notifies the user where the file has been printed or if it was not possible to print the file.

Fig. 3.
figure 3

Screens of (a) GREat Tour, (b) GREat Mute, and (c) GREat print

Full size image

These ubiquitous mobile applications (See Fig. 3) were evaluated regarding the following five quality characteristics presented in Sect. 2.1: context-awareness, mobility, transparency, attention, and calmness. Twenty-four measures were collected to evaluate these characteristics, some examples of such measures are presented in Table 1. More details about these evaluations can be found in [6, 7]. However, we present some examples of the results of these evaluations in Table 2.

Table 2. Results for UbiComp applications evaluations. Adapted from [6]
Full size table

The collection of these software measures makes possible to evaluate the quality of interaction in the applications. The results are an indication that the measures are capable of evaluating a ubiquitous application according to the selected HCI quality characteristics.

3.2 IoT Applications

Automa GREat is an IoT application developed to control airs conditioners and the lamps of the GREat Lab’s seminar room. The users can switch on/off the air conditioners and increase and decrease the temperature. About the lights, the users can turn on/off and set their color and intensity. There are two modes of user interaction with the application: manual and automatic. In the manual mode, the user controls all things, and in the automatic mode, the application uses the motion sensor to turn on/off the things. To manage data from/to devices, the application uses the middleware LoCCAM (Loosely Coupled Context Acquisition Middleware) [23] that is based on [24, 25, 33, 34].

GREat Room provides a way to register the presence of people in a particular room, making possible to see who is in that room at a given time and it can create a report about the presence of the persons. Furthermore, the primary requirement is the creation of the virtual group with these individuals to share files. The application uses the information from the beacons to identify new things and persons in the room.

Figure 4 presents the screens of these applications.

Fig. 4.
figure 4

(a) Screen of the automa GREat, and (b) Screen of the GREat room.

Full size image

These two IoT applications were also evaluated using the software measures for UbiComp applications presented in Sect. 2.1. We present some results of these two evaluations in Table 3. Details of the GREatRoom evaluation can be found in [5].

Table 3. Results of IoT applications evaluations. Adapted from [6]
Full size table

In general, these measures give us a comprehension into not only whether the app achieves the Human-Thing interaction or not, but also what can be improved in the application. For example, GREatRoom still needs to be improved regarding the adaptation degree (78%) and adaptation correctness (79%). These measures obtained bad results, because the application presented some problems when the user position is on the border between two environments, where the system cannot be sure which room the user should assign. Moreover, we discover that Automa GREat is not working regarding the context-awareness, which means the application cannot adapt itself when the user enters or leaves a room.

In other measures, we found results that diverge between both applications. For example, we have considerable differences in the “Availability Degree” and “Courtesy Degree”. The GREat Room receives the best evaluation, because it reacts correctly and in the better and calm way. Automa GREat, in turn, does not work well in the automatic mode for the sake of the inability of detecting the user presence. This leaves the result “low” in the measure that checks if the behavior changed was the better one. The value “medium” in the “Availability Degree” occurs, because the application does not perform the required functions. Users also consider that the application does not interact at the right moment, what leaves us with the result “medium” in “Context-awareness Timing Degree”.

4 Discussion

In this section, based on the applications’ evaluations, the literature, and our experience, we discuss how we can benefit from the UbiComp move towards IoT. Also, we detail the main differences between these two kind of applications’ evaluations, highlighting what was not detected by the measures from UbiComp applied in the case studies in the previous section. We used the following four open questions to organize our discussion.

What are the commonalities of HCI UbiComp evaluation that can be also used in IoT applications?

The characteristics and measures that come from UbiComp, which are presented in Sect. 2.1, are also useful to evaluate IoT applications and they give us a good insight into what can be improved in IoT applications as presented in Sect. 3. For example, GREatRoom needs to be improved regarding adaptation degree and adaptation correctness degree. Automa GREat, in turn, needs to be improved regarding all measures for the Calmness characteristic. Therefore, these case studies show an example that characteristics and measures for UbiComp are useful to IoT applications. Nevertheless, the evaluations presented in Sect. 3 indicate that:

  • The same UbiComp evaluation process can be used to IoT applications. In our work, we used the same process from usability testing with profile questionnaires, observation, and measures collections;

  • The necessary data collection for the measures calculation in IoT applications requires the same effort as in UbiComp applications. All IoT applications need to be instrumented like we do in UbiComp applications;

  • The questionnaires used to capture the subjective measures from users are relevant and suitable in IoT as well; and

  • The results from the measures in IoT can be interpreted following the same analysis procedure as in Ubicomp.

Our answer is that good part of what is used in Human-Things evaluation in UbiComp can be used in IoT. However, several problems arise from the Thing-Thing interaction that needs to be evaluated as pointed out in the next questions.

What interaction problems were not identified in the IoT applications by the UbiComp measures?

Regarding the case studies presented in the previous section, the collected measures were not able to identify several problems that the IoT applications presented. For example, GREatRoom presents problems when multiple mobile devices enter and exit a room, because it generates an inconsistency in the data presented in the application’s interface to different users. This is a problem of things being out of sync with the other things.

The other problem is the lack of conflicts handling. Conflicts occur among contrasting goals [37]. Despite being a subject already explored in the field of smart homes and ubiquitous computing, IoT brings greater complexity to this problem, because IoT environments allow remote control by users, automated rules programmed by users and context-aware decisions in the same system. For example, the other application, Automa GREat, has no conflicts handling. If two users send opposing requests, Automa GREat is not able to handle this conflict. Another problem regarding conflicts in IoT, not detected by the evaluations, is when a user may want to turn the lamp on remotely via smartphone, but the application wants to turn it off because no one is at home. Therefore, the measures from UbiComp are not able to evaluate if there are conflicts in Human-Thing and Thing-Thing interactions.

Another issue that was observed during the case study with Automa GREat was a delay when users turned on the air conditioner, which affects the system’s responsiveness. Also, the state of things (i.e., air conditioning and lights) were not being displayed properly in the interface, whenever the user changed the tab, the application loses the current state about the things.

In short, the main interaction problems that arise from IoT applications are related to the Thing-Thing interaction, which impact on the Human-Thing interaction. So, to answer this question taking into account the case studies, we should consider the following: synchrony, the lack of conflicts handling and communication delay between things.

What characteristics and measures of Thing-Thing interaction should be evaluated in IoT systems?

Based on the previously questions and literature review, we classify the characteristics that should be evaluated according to two levels: Internet and Thing. Internet-level is related to intrinsic characteristics of the communication between the things. The only work found, for now, that mentions characteristics for the Thing-Thing interaction is [41], which consider synchronicity, responsiveness, and reliability as significant issues to the interaction quality. These are discussed as follows.

  • Synchronicity: The things may be out of sync with other things. In GREatRoom, for example, the time it takes to synchrony the things can not be long because the application can show different information for different users that are in the same context [41]. In Automa GREat, the air conditioner may show a different temperature than what is displayed on the user’s device.

  • Responsiveness: There are no certainties about how fast it will be for two things to communicate with each other though the Internet. The result is that latency (the time it takes for a message to pass through the network) can be unpredictable, which affects the system responsiveness [41]. In Automa GREat, for example, it is important to measure the time it takes to turn on the air conditioner. This time will affect the user perception of the quality of the interaction.

  • Reliability: A command sent through Internet can fail to arrive. It is hard to guarantee that a user command will not get lost. It is not acceptable to have everyday objects failing about their main functions [41]. For example, in Automa GREat, no user would like to use application that fails to turn on lights and air conditioner. Then, it is important to measure, for example, how many failures occur and the time between failures.

Thing-level characteristics are related to components/characteristics that the things may have, and that can be very challenging for Thing-Thing interaction quality and, consequently, to the Human-Thing interaction quality. Battery, context-awareness, interoperability and difficulty of installation are important issues to the Thing-Thing interaction quality.

  • Battery: Some things need to be connected all the time, and this requires a lot of energy. It is a big issue manage the trade-off between network and battery. If a thing goes off-line because of battery, the other things will not be synchronized, and then, the synchrony will be affected [41].

  • Context-Awareness: The things must have the ability to correctly sensing the environment and the user [35]. If a single thing fails about the context, a chain of failures can happen between things that are connected with it.

  • Interoperability: We do not have a real Internet of Things today. In a true IoT, anything can connect to any other thing. However, today, many things are locked in proprietary ecosystems [41, 50].

  • Difficult of installation: The difficult of installation is about the process of connecting things with other things already existent. This process can be very difficult for a user, since they may use different network technologies [9, 19, 49].

Therefore, our answer is that software measures should be defined to evaluate the characteristics discussed above: Synchronicity, Responsiveness, Reliability, Battery, Context-Awareness, Interoperability and Difficult of installation.

What are the major challenges to the interaction in IoT applications?

The challenges are related to the increased complexity that an IoT system can have [41, 50]. It is possible to add several smart objects in an IoT environment, for example, a smart lock for a door, an intelligent air-conditioning, and a smart coffee. Each one can deliver a service to another. This brings several challenges for the interaction, such as interoperability, consistency of the interactions between several objects and interfaces, detection and resolution of conflicts, and the choice of techniques and methods to evaluate the interaction. These challenges are discussed as follows:

  • Interoperability: One of the main challenges of IoT lies into interoperability [41, 50, 53]. In a true IoT, things can connect with each other, independently of the manufacturer. Today, many IoT devices do not work with all other IoT devices [50]. People are using single IoT systems, for example, NestFootnote 5, Philips lampsFootnote 6, DanalockFootnote 7, SmartThingsFootnote 8. This represents a problem for user interaction because they will depend on the manufacturer. This lack of interoperability between things is important to designers because it creates idealized expectations from IoT that today cannot be achieved [41].

  • Consistency of the Interactions: An IoT system can have many interfaces (e.g., mobile phones, tablets, desktops) and many devices [50]. A recent study from [12] states that support for multi-device interactions has fallen behind users’ request to leverage the diverse capabilities of the devices that surround them. Thus, there is a difficulty in designing interactions between devices [21], especially in IoT, that one single system will have several components (e.g., mobile interfaces, things, gateways). The lack of tools and methods for testing multi-device user experiences is a research opportunity.

  • Verification of Interest Conflicts: an IoT system may have several autonomous rules, many services, many users with different preferences [22]. There is a high risk of users feeling annoyed and overwhelmed by conflicts of interests. Therefore, how to deal with instruction from several users, things and services is a challenge. A research opportunity is to detect or predict if new instructions and actions are in conflict with others.

  • Evaluation: There are several possible combinations of things, services, and interfaces that the user could experience, which makes the evaluation a challenge for IoT systems [41, 50]. Any combination can generate unexpected behavior and can impact in the user interaction. More research is necessary about how the evaluation has to be performed in such scenarios to efficiently discover user experience problems. Also, it may be necessary to adapt the traditional methods to take into account the Thing-Thing interaction issues previously cited. For example, the existing questionnaires of usability do not consider these issues [3]. Additionally, work from [13] believe that several new heuristics specific to IoT will arise with the spread of the field. We also can cite two more challenges for the evaluation, as follows.

    • Data Collection: With our experience in the IoT evaluation, we have also perceived the increase in the complexity of data collection. This occurs for the sake of the various devices presents in the environment. Then, we need to define an interoperable approach capable of accessing these things and capture the data to analyze and evaluate the Thing-Thing interaction.

    • Environment Configuration: Moreover, all the situations are important and influence in the data collection such as the environment configuration that contributes and influences in the result of the evaluation. For example, if we consider the infrared sensor embedded in a device in a position inadequate to communicate with another thing, we can have a problem in our data capture and, consequently, in our evaluation.

Therefore, to summarize our answer to the last question, we believe that the main challenges to improve the quality of Human-Thing and Thing-Thing interactions in IoT applications are: Interoperability, Consistency of the Interactions, Verification of Interest Conflicts, and Evaluation.

5 Conclusion

In this paper, we first showed a summary of the HCI quality evaluation in UbiComp and the essential elements of IoT. Then, the interaction evaluation results of three UbiComp applications and two IoT applications are presented. The goal was to identify commonalities and differences between the interaction evaluation of these systems using the same characteristics (i.e., context-awareness, mobility, transparency, attention, and calmness). Next, we discussed IoT and its similarities and differences from Ubicomp using four questions. The evaluations were the foundation for our answers to the open questions together with our experience in these areas and what we found in the literature.

So, we hope to motivate researchers to work on the several challenges that we presented for IoT systems interaction evaluation, especially focusing on the opportunities of the evaluation of Thing-Thing interaction. Moreover, it is important to investigate the commonalities with Ubicomp interaction evaluation and reuse/adapt them to the IoT needs.