Keywords

1 Introduction

The Web increases accessibility and removes geographical barriers [1]. As more people use the Internet for on-line transactions, the need for effective, efficient and user satisfying e-commerce interfaces becomes significant. Product reviews and ratings need to be easily understood by users. Therefore, issues such as usability of the presentation, accessibility and clarity contribute to the decision making of users. Good exemplars of good design would require fewer users ‘clicks’ and display transitions. The term “modality” refers to the use of human senses (e.g. vision, hearing, touch, smell and taste) [2]. Currently, commercial e-commerce interfaces predominantly use visual means to communicate information. This often results to a visual information overload. The introduction of the new web technologies (Web 2.0) facilitated additional capabilities for users. Users can easily publish opinions, beliefs and thoughts globally accessible by the social media. Ratings can be shared through social media in platforms such as Facebook and Twitter. This paper presents a brief overview of literature relating to e-commerce, social media, user reviews, emojis and multimodality. It also describes the experimental e-commerce platform developed, the experimental design, results and conclusions.

2 e-Commerce, Social Media and Multimodality

2.1 e-Commerce

e-commerce can be defined as “the use of the Internet and other networking technologies for conducting business transactions” [3]. An organisation is considered to be e-business functioning when most (or a significant proportion) of its business is delivered electronically. Exchanging information is often enabled through the use of information technology (IT) [4] and lowers the cost of exchanging information [5, 6]. Generally, e-commerce not just involving selling or buying products online but it extends the business process such as handling customer online queries, integrating payment from customers, promotion of product and services. e-commerce is an umbrella concept that integrates a wide range of existing and new applications [7]. The Web increases accessibility and defies geographical barriers [8]. This is the reason that firms across the world have implemented e-commerce. With the increasing number of companies available online, it is important to understand some of the utilisation drivers of one platform over another [9].

2.2 Social Media

Web 2.0 enables users to interact and freely share information online [10]. The Oxford English Dictionary [11] defines social media as “websites and applications that enable users to create and share content or to participate in social networking”. The ability to create content online creates an influence of one user over another in social media websites such as Facebook and Twitter.

A key business component of social media is that it allows consumers to evaluate product, make recommendations, and link current purchases to future purchases through status updates and Twitter feeds [12]. Social media is rapidly becoming one of the main sources for product reviews. Previous research investigated the role of social media on business. Lucas [12] conducted a study on the influence of social media in consumer purchasing behaviour and found that most of the participants purchased online based on social media previews. This study had 249 participants, 59 % where using Facebook and 34 % using Twitter as social media tool in order to obtain peer reviews. Social media is generally regarded as a useful aid to customers who seek advice or peer reviews of products. For example, consider the Trip Advisor platform, people make purchases based on previous experience and recommendation from unknown people. Several studies have focused on the influence of social media on potential customers. Boomer [13] discussed the influence of social media and its role. This becomes even more relevant as increasingly people tend to share most of their online experience using social media networks (e.g. Facebook and Twitter). These sources are exemplars of frequently used platforms to extract or retrieve product reviews.

2.3 User Reviews

Social media appears to become the platform users rely on to get reviews. When users are sharing reviews regarding a product or service they are directly influencing the purchase decisions of other users. According to [33] consumers evaluate product information (e.g., product reviews) in order to achieve their consumption goals. Moreover, electronic word-of-mouth (eWOM) is created when users share information and their experiences online [34]. The eWOM is likely to be much more powerful than WOM as it could potentially reach unlimited number of users [35]. It is a powerful product information source [36]. User reviews are usually categorised as positive, neutral or negative. Positive product reviews provide information about satisfactory experiences with the product, and thus represent opportunities to attain positive outcomes [33].

A study focused on tripadvisor.com conducted by Vermeulen and Seegers [37] found that positive hotel reviews improve the perception of future customers. They concluded that exposure to any (positive or negative) hotel reviews increases hotel awareness, especially if the hotel is less known. A similar study conducted by Ye et al. [38] suggested that positive hotel reviews result to more bookings. Reviews being shared among other users or groups have different value to other users depending on the content of review and emotional context. For example, consumers that attribute negative emotions to the reviewer’s personal dispositions rather than the product, those emotions are unlikely to influence other users [36].

2.4 Multimodality

Several user interfaces communicate reviews using text with some graphics to communicate product reviews. Although this is considered acceptable to users, it could result in overloading the users with textual information [14, 15]. When presenting consulting user reviews and ratings, other non-textual means can be used to aid efficiency, effectiveness and user satisfaction. For example, multimodal interaction for product reviews and ratings may help to browse large volumes of this data easier [16]. Multimodal applications may use non-speech sound, text and hypertext, animation and video, speech, handwriting, gestures and computer vision.

Combining visual and auditory metaphors enhances the user experience. The auditory metaphors consist of recorded speech, earcons and auditory icons. The more metaphors used, the greater the volume of information that can be communicated. Avatars often incorporate the use of speech and human-like animated facial expressions and body gestures [17]. For instance, in e-learning interfaces, multimodality has shown to be useful in enhancing the usability and users learning performance [18]. Previous studies suggest that the use of more metaphors, including graphics, often enables users to perform tasks faster. For instance, a study by Rigas and Memery [19] showed that multimedia helped users to learn more material than a typical text-and-graphics approach. Users also performed different tasks more successfully. Another study [20] showed that multimedia metaphors helped users to make fewer mistakes in intermediate and complex tasks. In some cases, the time taken to complete tasks was also reduced. This shows the importance of multimedia or multimodality in making tasks executions easier. Another study by Rigas and Memery [21] showed that multimedia helped users to learn more material than using text-and-graphics media and assisted them to perform tasks more successfully. This shows the importance of multimodality.

Multimodal systems have been developed to support functions such as increase system accessibility for diverse users [22]. Avatar is a computer-based character that has been utilised to virtually represent one party in an interactive context [23] using speech, facial expression or body gestures [32]. Avatars are often used as a tool to support e-learning environments [18]. As multimodality refers to the use of different communication channels, avatars are considered as an additional metaphor to improve the visibility and communication aspects of any system. Avatars can be classified into three groups. These are abstract, realistic and naturalistic [24].

3 Experimental Platform

An experimental e-commerce platform was developed to act as basis for this empirical investigation. Two interfaces were designed. These were a text-based with emojis and a multimodal with expressive avatars. Both interface versions communicated the same information relating to the reviews and ratings of products. All the reviews were sourced from Facebook and Twitter. The presentations were designed to deliver the same information about the products being displayed. This information included different type of reviews (positive or negative) along with ratings score (1 to 5). The products used in this platform were laptops with different specifications, prices and ratings. Figure 1 shows the conceptual model for this experiment.

Fig. 1.
figure 1

Conceptual model showing the relationship between social media reviews messages and multimodal metaphors.

Full size image

The presented content included three sections the product, product specification and the reviews. The complexity of the reviews communicated increased in every task. Table 1 shows the mapping allocation of the multimodal metaphors to the information communicated. Customer review comments were communicated using text and the different ratings using text and multimodal metaphors. Guidelines for multimodal information presentation [22] were followed for the development of the interfaces. The expressive avatars used in the experiment had facial expressions correlating to the review rating (positive, negative or neutral). These followed established guidelines in the literature [25]. Figure 2 shows the facial expressions used in the expressive avatars.

Table 1. The allocation of metaphors to the information communicated
Full size table
Fig. 2.
figure 2

Facial expressions used in the expressive avatars

Full size image

4 Experiment

The experiment measured efficiency, effectiveness and user satisfaction. Each user was presented with different presentations of reviews and ratings using four experimental tasks. The user sample consisted of 24 users that were opportunistically divided into two groups. The independent variable was the method used to communicate reviews and ratings. This variable has two versions. These were the text-based with emojis and an avatar-based design. The experiment consisted of four parts:

  1. 1.

    Pre-experimental questionnaire.

  2. 2.

    Perform tasks.

  3. 3.

    Post-task questions.

  4. 4.

    Post-experimental questions.

4.1 Procedure

For consistency throughout the experiment, the same procedure was applied for the two groups. The experiment started by requesting users to answer a pre-experimental questions that aimed to collect data relating to:

  1. 1.

    General user profile (e.g. age, gender and education).

  2. 2.

    Previous experience with computers, Internet and online shopping.

  3. 3.

    The frequency of use of social media networks.

During the pre-experimental questionnaire, users were presented with a video tutorial for five minutes. The tutorial presented an introduction to the experimental e-commerce platform. Two tutorial videos were used (one for each group of users). The control group was presented with a tutorial demonstrating the interface with the text-based reviews and emojis. The experimental group was presented with the facial expression avatar-based interface. The link between facial-expressions and the ratings of the reviews was also highlighted in the tutorials. Thereafter, the users in both groups performed the same tasks (i.e. four tasks) but using the different interface according to the group (i.e. control and experimental). On completion of the experimental tasks, users answered a user satisfaction questionnaire.

4.2 Experimental Proposition and Hypotheses

The experimental proposition is that multi-modal metaphors and facially expressive avatars will provide more effective, efficient and user satisfying presentations of social media based product reviews in an e-commerce platform compared to a textual based with emojis approach. This paper focuses on the effectiveness and efficiency parameters of the above proposition. The hypotheses are:

  1. 1.

    Presentations of reviews and ratings using facially expressive avatars will be more effective than text-based with emojis in terms of tasks completed successfully.

  2. 2.

    Presentations of ratings using facially expressive avatars will be more efficient than the text-based with emojis in terms of users’ products comparisons.

  3. 3.

    Presentations of ratings using facially expressive avatars will have the same efficiency in user-based comparisons for up to four products.

4.3 Tasks

Each group performed the same four tasks but with the corresponding interface. Previous studies showed that the metaphor affect user performance depended on the level [20, 25] and the type of a task being examined [26, 27]. Hence, the tasks were designed to follow the same procedures as previous experiments. The level of difficulty in each task was gradually increased in a way that the first task was the simplest and the fourth task was the most complex. Therefore, the tasks were classified into easy, moderate and difficult. Each task had a set of requirements. Users had to choose the correct product based on criteria based on the reviews. The actual reviews varied from one task to another. For instance, for the Group 1 and task 1, the reviews were simply presented one-by-one (one product review presentation) but for task 2, reviews of products were presented together so as the user could compare products. In the multimodal presentation, as the user proceeded from one task to another, the complexity of the review rating also increased. Therefore, the more difficult the task, the more information was communicated. The complex tasks contained larger volumes of information compared to the easy and moderate tasks. On completion of each task, users answered questions based on that task. The aim of these questions was to evaluate the performance of the user based on the information and the review interface presented by the task.

4.4 Sample

The user sample consisted of 24 users who had no prior exposure to the experimental platform. All users had no prior knowledge on the multimodality metaphors as used in the experiment. They were requested to perform all tasks and answer all questionnaires. These 24 users were opportunistically assigned to two groups (n = 12) to evaluate the two conditions (i.e. text-based with emojis and facially expressive avatars). This volume of sample is considered to be sufficient for this usability evaluation [28].

5 Results and Discussion

The results were analysed in terms of the time taken by users to complete the tasks (efficiency), number of correct selections of products (effectiveness). Inferential statistics were used to examine the difference between variables [29]. When the data was not normally distributed, the Mann-Whitney t-test was used [30]. The mean, median and mode were used to perform the statistical analysis. Also, Kolmogorov-Smirnov test [31] was used in the statistical analysis to test, calculate and present the normal distribution of experimental results. The statistical analysis used α = .05 and the significance using p-value = .05 (which refers to be less than 0.05).

Fig. 3.
figure 3

Users’ profile in terms of age, gender, and education level in the three groups

Full size image

5.1 Profile of the Sample

Pre-experimental questionnaires were used to collect the users’ profile information (e.g. gender, age, education or prior experience). Figure 3 presents the profile of the sample. The control group (text-based with emojis) consisted of 58.33 % of users aged between 25 and 34, 36.11 % between 18 and 24 and 5.56 % between 35 and 44. The second group (multimodal) had an age range of 58.33 % between 25 and 34, 36.34 % between 18 and 24 and 5.33 % between 35 and 44. The gender distribution was 25 % females and 75 % males in the control group and 16.67 % females and 83.33 % males in the experimental group. The education level of the sample was predominantly undergraduates. Figure 4 shows the sample’s prior experience relating to the use of computers and the Internet. The data suggests that the sample was knowledgeable and experienced. This was expected given that the sample was drawn from University students.

Fig. 4.
figure 4

Users’ knowledge in terms of using computers, Internet and online shopping frequency

Full size image

5.2 Effectiveness

The effectiveness was measured by the frequency of tasks completed successfully by users during the experiment. This measure was considered for all the tasks in total and for task complexity (easy, moderate, and difficult). Users in the experimental group performed marginally better than the users in the control group (Fig. 5).

Fig. 5.
figure 5

Percentage of correctly completed tasks achieved by the users in the two groups for all tasks (A) and for task complexity (B).

Full size image
Fig. 6.
figure 6

Total number of successful completed tasks by each user

Full size image

The total number of correctly completed tasks in the experimental group (facially expressive avatars) was 43 compared to the 37 for the control group (text-based with emojis). The mean value of successfully completed tasks per user for the experimental and control groups was 3.58 and 3.08 respectively. This experimental group demonstrates marginal improvement of successfully completed tasks compared to the control group. The ANOVA test results showed that that the variance was not significant (p at 0.05). The experimental group outperformed the control group in the easy tasks only. The experimental group had 87 %, 75 % and 66.7 % completion rate for easy, moderate and difficult tasks respectively. The control group had 83 %, 75 % and 67 %. Although these results do not show that one method is better than the other in terms of efficiency, there is a prima facie case that in principle the application of expressive avatars does not hinder the effectiveness of users. This results points to the fact that the two approaches can be collaboratively employed to communicate reviews and ratings. The emojis also helped users to quickly browse reviews that in turn accelerated the completion of tasks. Figure 6 shows the total number of successfully completed tasks achieved by each user for both groups. Results show that 4 users successfully completed all tasks in the control group (users 1, 2, 3 and 6) while 3 users in experimental group (users 8, 9 and 12). The mean value of tasks completed successfully per user was 3.08 and 3.17 for the control and experimental groups respectively.

6 Conclusion and Future Work

The variance between the control and experimental groups was not significantly different in terms of tasks completed successfully. This indicates that both approaches taken to communicate reviews and ratings are valid. It is important to investigate further under different task circumstances. The use of emojis shown to be particular useful. Users obtained an overall viewpoint relating to the review and rating (positive, negative or somewhere in between) at a glance. This makes a strong prima facie case for the use of emojis as additional element of the review and rating entry. There is a need to understand the way in which emojis and expressive avatars can be combined to communicate larger volumes of data.