1 Introduction

Nowadays, the use of the Internet has become a critical source of productivity for any company because of its capability to extend the information to many users around the world. Accompanied by the continuous growth and high adoption of this technology, Web software applications have emerged as a means of electronic commerce of products and services.

However, due to a large number of Web sites that daily appear in this competitive market, companies are no longer concerned only with their presence on the Internet. Companies have been forced to change their strategy by focusing on the development of high-quality software products, which meet certain properties related to the quality of use of the systems [4]. One of these properties is the communicability, which is defined within the framework of the Semiotic Engineering as the feature of software to effectively and efficiently conveys to users its underlying design intent and interactive principles.

The Communicability Evaluation Method (CEM) is a technique to measure the level of communicability of a software application. The aim of this method is to determine the extent to which users achieve to understand the design of a graphical interface [8]. From the approach of the Semiotic Engineering, the Human-Computer Interaction (HCI) is perceived as a communication between users and designers, given that developers manage to communicate different ways to achieve specific goals through the design of the interface [1, 2]. These messages can be both explicit (via texts), or implicit (via icons, symbols, graphics, patterns, frames, and other forms of design). This method allow designers to appreciate how well users are getting the intended messages across the interface by identifying the communication breakdowns that take place during the interaction [9].

This research is the result of an assessment based on the Communicability Evaluation Method (CEM) conducted in the Web domain. The software system that was selected to perform this study was Booking.com, a transactional Web site that is used for booking hotels worldwide. As a result, it has been possible to identify the communication breakdowns that could arise during the interaction between the user and the system. This case study has allowed defining the degree of communicability of the software application and the quality of metacommunication between designer and user for future improvements of the Website.

2 Communicability Evaluation Method

In order to conduct an evaluation of the user interface design from the perspective of the Semiotic Engineering, we used the Communicability Evaluation Method (CEM) proposed by de Souza [11]. CEM is a method to determine the degree in which the system designers achieve to communicate users their design intents through the interface design. As well as other theories that were inspired in the Semiotic Engineering, CEM states that the human-computer interaction is a particular type of communication between humans which is mediated through a computer [5]. This interaction involves the participation of both designers and users, and the communication occurs at the time in which users interact with the system interface. If the purpose of each design element is properly communicated, users will be able to achieve their goals through the use of the system [10].

The Communicability Evaluation Method (CEM) is a technique in which a representative number of end users are requested to interact with the software product to be tested. This interaction is guided by a predefined set of tasks that users must perform during the software evaluation. All interactions must be recorded using devices and tools to capture the user’s screen and the user’s face, in order to subsequently analyze the mouse movements and the facial expressions. This method includes three phases [3]: (1) Tagging, (2) Interpretation and (3) Semiotic Profile. The process of CEM is illustrated in Fig. 1.

Fig. 1.
figure 1

The three core steps of SEM [11]

Full size image

In the first phase, the videos, where the interactions were captured, must be analyzed. The purpose is to identify events that are indicative of breakdowns in the communication. There are thirteen expressions of communicative breakdown to classify each event. These tags represent the specialist’s interpretation of the user behavior. Once all events are labeled, the evaluators must interpret the meaning of the whole set of communicative breakdowns that were detected. For this phase, the experts should consider the frequency of each tag and identify the design problems that cause these breakdowns. Finally, in the last phase, specialists must elaborate the original message of the designer. The classification of the communicability problems helps specialists to achieve an in-depth characterization of meta-communication, the meaning of the overall designer-to-user message [7].

2.1 Tagging

In this phase, the specialists should use the recordings to analyze the user’s reactions and identify communicative breakdowns. Evaluators must relate each evidence of communicative breakdown to 1 of the 13 tags proposed by the Semiotic Engineering. These tags are written in natural language, and they are sometimes expressed verbally by users during the test. According to de Souza [11], this process can be described as “putting words in the user’s mouth”. The thirteen tags that characterize communicative breakdowns between user and designer are [7]:

  • “I give up.” The user is unable to achieve the proposed goal. There are not enough resources such as time, will, and patience to reach it.

  • “Looks fine to me.” The user is unable to realize that the outcome is not the expected one. The result seems to be fine for the user. However the task has only been partially completed.

  • “Thanks, but no, thanks.” There is enough evidence to ensure that the user knows the design intentions, however, decides to follow a different path. The user declines designer’s invitation to engage in a particular kind of communication.

  • “I can do otherwise.” The user is not aware of the design intentions. Therefore, he does something different from what is expected. The interaction reveals that the user has not received the designer’s message about how to use the system in the particular context in which he is.

  • “Where is it?” The user is looking for a specific functionality of the system, but is unable to find it. The user is expecting certain sign or element in the interface design to carry out a particular strategy, however, can not easily identify it.

  • “What happened?” The user repeats an operation because he is apparently unable to assign meaning to the result that is provided by the system. The user can not identify the effects caused by their actions (halt for a moment).

  • “What now?” The user does not know what to do next, and performs a random action. He is clearly looking for a clue that brings him back into the right path to achieve the proposed goals.

  • “Where am I?” The user performs actions that are inappropriate in the context in which he is. Although the design elements are successfully identified, these are used in a wrong context.

  • “Oops!” The user chooses certain system functions by mistake and try to leave immediately the unwanted state (commonly via the command ’Undo’ or by attempting to restore some previous states of the system).

  • “I can’t do it this way.” The user realizes that the sequence of actions he is performing does not lead to the goal. Therefore, he abandons this path of interactions (composed of many steps) and chooses a different one.

  • “What is this?” The user explores some possibilities of interaction to acquire knowledge about the system functions. He examines the interface design in order to find explanatory tips or clues about the meaning of the elements.

  • “Help!” The user accesses the help system. He deliberately calls this function by pressing F1 or by reading the manuals that are available. The online support is also considered as a source of help.

  • “Why doesn’t it?” The user performs a series of actions expecting a specific result from the system, however, this scenario is not achieved. Although the outcomes are explicit and clear, these do not match with the user’s goals. The user repeats his actions again to determine if he is doing something wrong. He reviews every step carefully because he is sure of the chosen path.

2.2 Interpretation

In this phase, the specialists must analyze the collected data to identify all the problems of the graphical user interface. Each communicative breakdown is related to a specific category of issue [7]: navigation, meaning assignment, task accomplishment and declination/missing of affordance.

2.3 Semiotic Profile

In this final phase, evaluators will be able to identify the meaning of the overall message between users and designers. After an in-depth analysis of the communicative breakdowns, specialists will attempt to retrieve the original designer’s metacommunication. They will assume the position of the designers by answering the following questions in first person [7]:

  • Who do I think are the users of the software product I have designed? The purpose of this question is to identify the main characteristics of users (the listeners of the designer’s message).

  • What the users want and need? This question allows specialists to identify the mismatches between designers and users (the differences between the design intentions and what the user is actually expecting from the design).

  • What are the users’ preferences according to what they want and need? Why? The purpose of this question is to identify the designer’s justification for the interface design that was proposed. The answer to this question allows specialists to determine whether the design decisions are consistent with the user expectations.

  • How is the system I have designed for these specific users, and how they should use the system? The purpose of these questions is to identify the degree in which the expressions and content of the designer’s metacommunication are being conveying to users through the interface design.

  • What is my design vision? Finally, this question allows specialists to identify the degree in which the design is properly understood (and accepted) by the user.

3 Research Design

3.1 Participants

All the participants were students at the National University “Pedro Ruiz Gallo” located in Perú. In order to conduct this experimental case study, four students were randomly chosen from an elective course called “Usability Engineering”, that is shared by two academic programs: An Undergraduate and a Master’s Program in Computing. They aged between 24 and 41 years old, and three of them had previous experience using the software. We used the following IDs to represent each participant: P1, P2, P3 and P4.

3.2 Description of the Software System

The software system that was evaluated is Booking.com, a transactional Web application for hotel booking. The goal of Booking.com is to offer the best prices for any type of accommodation, from small, family-run beds and breakfasts to executive apartments and five-star luxury suites [12]. One of the expectations of this trademark is to provide a usable Web site, in which the user easily finds all information regarding hotels. This system should allow travelers to choose from a variety of accommodations around the world in an easy and efficient way.

3.3 Test Design

The communicability evaluation test involved two activities. Participants had to perform these tasks consecutively and in the established order. Moreover, there was a specific time to complete each task. The design of these activities was based on a previous study [6], in which we conducted a heuristic evaluation to a similar software product in order to identify the most critical usability issues of this type of application. The issues that were considered for the development of this test are:

  • The lack of visibility of certain important elements of the interface design such as search filters, features and services offered by hotels, and information related to accommodations.

  • The lack of certain options of the system, which allow users to make better decisions. For instance, the lack of a tool for comparison among the alternatives of hotel accommodations that are offered by the Web site, and the absence of a feature to sort the results based on user criteria.

  • The ambiguity in the meaning of certain icons and symbols that are used as part of the interface design, and the confusion these representations can cause among the users of the system.

  • The lack of a cultural perspective for the development of the interface design. Given that a Web site is used worldwide, some cultural aspects that should be considered such as language, currency, date formats, writing direction, special characters, calendars, symbols, colors, etc.

Given that the purpose of this test is not to verify the usability of the Web site but the communicative breakdowns between the user and the designer, the number of usability issues that were considered for the test design are appropriate. Attempts were made to develop a realistic scenario, a real situation with possible conditions that could arise when users book a hotel. The scenario that was considered for the test of communicability is described as follows:

“You have decided to spend your summer vacation in Europe with your best friend. After a long talk with your partner, you agreed to be responsible for all costs of accommodation. However, you only have S/.2400.00 (local currency of Perú) for this purpose. Also, you have considered that a three-star hotel is acceptable for both. Given these conditions, please complete the activity, considering that the accommodation date is from February 4 to February 10, 2016.”

In the second part of the test, the users were requested to search for a particular hotel. The following scenario was proposed to evaluate the ease with which the users manages to determine if given a specific hotel name, they achieve to conclude if it meets their preferences. Both activities are described in Table 1.

“Some friends have recommended you the following hotel located in Paris (France): Jean Gabriel. For this reason, you will review its services, features and rates to determine whether it meets the above conditions.”

Table 1. Description of tasks of the communicability evaluation
Full size table

3.4 Test Environment

The communicability evaluation of Booking.com was performed on January 23th, 2016. This study was conducted in a computer lab at the National University “Pedro Ruiz Gallo”. We used the following electronic devices and software:

  • A webcam to record all gestures and facial expressions of the users.

  • Cam Studio Open Source, a free streaming video software to record the interaction between user and system.

  • TeamViewer, a software package for remote control and desktop sharing.

4 Data Analysis and Results

4.1 Tagging

After an analysis of the interaction between user and system during the test and a study of the recorded videos, we identified the communicative breakdowns. The results show that no events were registered for the following communicative breakdowns: What happened?, What now?, Where am I?, Oops! and Help! The events for the remaining tags are described as follows:

I Give up.

  • After several attempts to display the rate per night of the hotel room, P1 gave up this activity. After a brief interview, he said he was spending a lot of time and chose to continue with the other activities.

Looks Fine to Me.

  • P2 and P4 tried to find quickly the hotel that meets the proposed conditions by using the search filter. However, this tool was used in a wrong way. Nevertheless, the results of these actions seem to be fine for them. Instead of helping them, these actions complicated the search, and they had to spend more time than the other participants.

Thanks, but No, Thanks.

  • The system displayed a calendar in which P2 could specify the check-in and check-out dates. However, he decided to use another functionality (implemented by a combo box) to establish these dates. He is aware that both choices lead to the same result, but prefers to leave the current workflow to use a different option.

  • There is an option in the interface to search by hotel name. However, P4 decided to use search filters to identify the hotel that was requested in the test.

I Can Do It Otherwise.

  • There is not an option in the system to display the room rate per night, only the total price. Therefore, P2, P3 and P4 decided to use the calculator of Microsoft Windows to perform this estimation.

Where Is It?

  • P2 tries to locate an option that allows displaying only those hotels whose total price ranges from a certain amount of money to the available budget.

  • P2 tries to find a functionality that allows sorting the search results by room rate from the cheapest to the most expensive hotels.

  • P3 tries to find a link to visualize the services that are offered by the hotels.

  • P2 and P4 try to find an option that allows them to view the room rates per night of the selected hotels.

  • P1 tries to locate the option that allows performing a search by hotel name.

I Can’t Do It This Way.

  • P2 and P4 selected some search filters to find a hotel that meets the proposed conditions of the activity. However, this action delayed the search process instead of simplifying it. Users concluded it was not the proper way to perform the activity. Therefore, they stop the current sequence of actions to choose a different path.

What Is This?

  • P1, P2, P3 and P4 tried to find out how the search filters should be used. They tried to determine if these tools are useful and appropriate to achieve the goals of the task.

  • P2 and P4 examined some system options to sort the search results. Users tried to determine if these options were appropriate to display the list of hotels in a specific order.

  • P1, P2 and P3 tried to discover the purpose of some elements of the interface randomly. After a brief interview with the participants, they stated the system provides many options whose intention is unclear.

  • P1 tried to find out how to use the system options that allow users to view images and videos about the hotels.

  • P2 examined some system options that allow users to modify the room type of a hotel booking, in an attempt to determine if there was a possibility to cover all the expenses with the proposed budget.

Why Doesn’t It?

  • P1 tries to perform a search using an element that was designed for another purpose. The results are not the expected by the user who is confused.

4.2 Interpretation

Given that the results show a small amount of communicative breakdowns, we can conclude that the level of communicability of Booking.com is acceptable. However, it is important to consider that three participants of this evaluation had experience using this software. Additionally, the test design was based on usability issues that were identified in a similar system. Although the test is valid, the tasks that were requested only addressed overall aspects.

The communicative breakdowns with the highest occurrence were: I can do otherwise, Where is it?, and What is this? These results establish that the system provides several options for users to identify immediately a hotel that meets their preferences. However, there are elements that are not visible enough and whose design intents are not explicit. The Website has been designed for users with experience in the use of this type of software applications. Novice users are forced to examine the interface continuously (What is this?) and determine how to use the system. Several times, they tried to find the location of certain options to achieve their goal (Where is it?) and performed a different path of actions to the proposed one by the designers (I can do otherwise).

4.3 Semiotic Profile

Who Do I Think Are the Users of the Software Product I Have Designed? Users are people who have experience in the use of software systems. They manage to become quickly familiar with my system. I should consider that they have different needs to choose a hotel. They have different types of perceptions and are from different cultures.

What the Users Want and Need? Users want to make a hotel reservation. However, for users to make a decision, they need to be informed properly about the hotel services, room facilities and rates. I am aware that users have different perspectives, and therefore, they could have different search criteria.

What Are the Users’ Preferences According to What They Want and Need? Why? Users need to find quickly a hotel that meets their needs. I believe that most users are looking for an affordable hotel, that is visually appealing and offers the essential services. However, I am also aware that all users have not the same preferences. Some of them might be considering specific criteria, such as if the hotel is near to a particular location, if the room rates are affordable or if the star rating is appropriate.

How Is the System I Have Designed for These Specific Users, and How They Should Use the System? The system I have designed offers all the information you need to book a hotel that meets your preferences. The software is flexible and implements multiple search filters. The search results will include a summary of the most relevant information about each hotel. The system will display photos and videos of the hotels for you to make a better decision. All information will be displayed in your language, and the rates will be in your local currency. You can sort the search results according to your preferences. Additionally, there will be an interactive map, in which you will be able to see the specific location of the hotels. I know that your opinion matters, and for that reason, I have designed a section where you can register your opinion as feedback for other users. Finally, I guarantee your privacy by establishing user accounts to register all your personal information.

What Is My Design Vision? Booking.com does not consider novice users. Many system options are not visible or explicit. Users who are not familiar with the software application are forced to discover how the system should be used. The level of communicability is acceptable, but there are still elements of the design that should be improved. Users do not follow the normal workflow that was proposed by the designers. However, the Website can be very useful for experienced users in the use of this type of applications. This technological tool offers multiple search options and provides the required information about each hotel. In this way, all users can make a decision according to what they are considering.

5 Conclusions and Future Works

This paper establishes the results of a case study, in which the Communicability Evaluation Method (CEM) was applied to a transactional Web application for hotel reservations (Booking.com). This inspection, whose purpose was essentially academic, involved the participation of four university students. The purpose of this assessment was to determine the degree in which the designers achieve to communicate users their design intents through the system interface. An analysis of the results allowed specialists to conclude that the level of communicability of this software application is acceptable.

Through the Communicability Evaluation Method (CEM) is possible to identify important aspects of the metacommunication between designers and users of a software product. In this study, we determined some communicative breakdowns in the interaction that delay the execution of specific tasks. The results of this research showed that the level of communicability of Booking.com is quite acceptable due to the small number of communication breakdowns that were identified. However, certain aspects need to be improved as the visibility and location of some design elements. The system should be more explicit with certain options since that users are continuously exploring the system to determine the result of certain features.

Most case studies that have been conducted in this area are focused on the evaluation of desktop applications. It is important to determine the results of the Communicability Evaluation Method (CEM) in other contexts such as mobile applications, video games, augmented reality applications and virtual worlds. Specialists should determine if it is necessary a change in the methodology to obtain accuracy results when a new category of software product is evaluated.