Keywords

1 Introduction

Augmented reality (AR) is a technology that integrates computer-generated digital information into the physical world using real-time computing techniques to provide users with an interactive experience of a combined virtual and physical world. In addition, AR provides experiences and sensations that cannot be perceived or imagined in normal natural environments by enhancing users’ sensory perceptions (e.g., visual, auditory or tactile perception) using digital technologies [1]. With the rapid popularization of smart phones and tablet computers, the video cameras, graphics processing capabilities, wireless communication functions and global positioning system (GPS) capabilities with which these smart systems are equipped have significantly improved the usability and practicality of AR technology. In other words, a user’s personal smart mobile device, which acts as another set of visual and auditory sense organs, can perceive virtual information embedded in the real world and integrate this information into the interface that is displayed to the user. When the user aims the camera on his/her mobile device at an object or a marker, the virtual information is directly superimposed and displayed on the user interface (UI) via the camera’s lens, and the user can interact with the system through a new UI that integrates virtual and real information [2, 3]. As a result, experiential marketing combined with AR technology has become the model application of this technology that is currently the most popular. An introduction by means of AR technology can not only can provide consumers with effective service experiences but can also improve brand value and consolidate long-term customer loyalty [4]. For example, the Japanese personal care company Shiseido allows consumers to experience different makeup effects in real time using the Magic Makeup Mirror makeup simulator [5]. The German shoe chain Goertz allows consumers to try on different styles of shoes virtually using Kinect motion-sensing technology. Consumers can upload photographs to Facebook and make purchases directly using Quick Response codes displayed on the monitor [6]. IKEA’s AR catalog, which is based on a similar concept, allows consumers to directly bring various types of virtual exhibits home and to easily try to match various furniture styles at home using markers in the AR catalog without having to move any furniture at home. These practical applications demonstrate that the use of AR technology has been gradually expanding from exhibitions and physical stores to personalized mobile experiences.

Good systems cannot be created using advanced hardware and technology alone. Mobile AR (MAR) applications can create more realistic and innovative experiences, but they also cause MAR services to face more usability-related challenges. However, when mobile applications and AR technology are being used and developed extensively, the focus is often on technological development, and the user experience and usability are not considered. As a result, there are few MAR applications on the market that are user-friendly and satisfy users’ expectations [7]. Therefore, to optimize the usability of AR applications and provide user-friendly interactive experiences, the present study first investigated the usability of mobile applications and AR technology. Then, based on suitable AR UI design usability principles, the present study designed a MAR application that satisfied the requirements of product marketing and interactive experiences. Finally, the present study measured the usability of the UI based on a mobile-specific heuristic evaluation checklist.

2 Usability Issues for MAR Applications

MAR allows the real world to become part of the overall interactive UI. Therefore, the timeliness of the display of virtual information, the way the information is visualized, the accuracy of marker recognition and the ability to interact with the user all affect the overall interactive experience of MAR [8]. Ko et al. [9] summarized the usability issues that a user faces when operating an MAR application. (a) Small display size – Interactions with MAR applications are based on mobile devices. Therefore, the size of the screen of the mobile device creates a usability issue when the user receives information. In addition, AR displays real and virtual information simultaneously. Therefore, it is necessary to avoid displaying an excessive amount of information and requiring complex UI operations when the UI is being designed. (b) Multimodal interface – MAR discovers virtual information in the real environment through a mobile device. Therefore, it is necessary to consider the properties and usability of two types of UI: tangible UIs and conventional graphical UIs. (c) Limited manipulation – The touch screen of a mobile device is its main input terminal. Gesture operations can generate a direct link between the user and a virtual object. In addition, suitable gesture operations can improve operability and the understanding of the information. On this basis, Ko et al. [9] proposed five usability principles for AR applications in a smart phone environment. (a) User-information – MAR applications need to provide users with suitable visual information and a clearly classified menu structure. In addition, MAR applications need to conform to users’ expectations in how they express information; (b) User-cognition – MAR applications need to be able to minimize users’ memory loads, allow users to react as expected and enable users to learn applications easily; (c) User-support – MAR applications should provide users with useful information, reduce errors, handle tasks and perform personalizing tasks; (d) User-interaction – MAR applications focus on the interaction between the user and the application, e.g., MAR applications provide feedback with minimum manipulation; and (e) User-usage – MAR applications emphasize actual usage, i.e., operations that are suitable for the situation and the degree of freedom of operation. These five usability principles all have corresponding evaluation conditions. The present study first designed an MAR application with the MAR usability principles comprising the design rationale and then measured the usability of this MAR application.

3 Design and Development of an MAR Application that Is Aligned with the Usability Principles

In view of the current lack of user-friendly MAR applications, the present study designed an MAR application, the Sakura three-dimensional (3D) MAR application, that conforms to Ko et al.’s MAR principles in terms of usability and the user experience. The Sakura 3D MAR application was jointly developed by the Digital Media Lab and WIDE Lab of Chang Gung University in Taiwan and the Metal Industries Research and Development Center in Taiwan to help Taiwan’s Sakura Corporation, a professional manufacturer of kitchen appliances and systems, break out of traditional marketing strategies and create a more realistic interactive experience for consumers using AR technology and thereby improve the benefits of marketing products and the quality of service. The MAR application designed in the present study supported all the smart phones and tablet computers that run the Android system (version 4.4 or later) and was released in the Android Market (Google Play) (https://play.google.com/store/apps/details?id=com.ming.SAKURA3D&hl=en) for consumers to download and use in September 2015. The main design functions of the Sakura 3D MAR application are discussed in detail in the following sections.

3.1 Virtual Information Display

To allow consumers to fully understand the form and external design details of a newly launched exhaust hood, the exhaust hood is displayed in the form of a 3D virtual model. Users can freely and directly operate this 3D virtual model using designed gestures (Fig. 1). Users can rotate this product model freely through 360˚ and enlarge and shrink it using two fingers to see its details. When the red arrow button on the outer ring is tapped, the corresponding product function and its innovative technology are displayed. When the illustration button is triggered, it becomes semi-transparent to provide users with visual cues during its operation. It is worth mentioning that the circular operation of the UI helps guide users to the proper locations and that each of the four triangular buttons points to information on one of the four major features. When a triangular button is tapped, the outer ring rotates back to the corresponding location to guide the user to a new position.

Fig. 1.
figure 1

Design of the guided interactive interface

Full size image

3.2 Brochure and the AR Marker

Traditional marketing is based on paper or digital catalogs or brochures, which do not allow consumers to easily imagine what a product will look like in their homes. An AR brochure is a physical paper brochure with augmented content, which is used as an AR marker. A user can display a virtual product in a real space (e.g., the consumer’s home) by aiming his/her smart phone at the number of the product in the paper brochure. In addition, size is often an important consideration in cookware purchases. The MAR application designed in the present study provides a one-to-one product experience (Fig. 2). Before purchasing cookware, consumers can use the paper catalog as an AR anchor point and the size of the catalog as a reference to display the actual product in real space on a one-to-one scale. AR catalogs allow consumers to visualize the size of a product in their homes and provides them with more diverse and convenient ways of shopping.

Fig. 2.
figure 2

One-to-one experience of different product models

Full size image

3.3 User-Friendly Interface Design

The user-friendly interface allows users to easily obtain the most important information through the UI’s layout and to understand the available options and the connections between different functions. Because the MAR application designed in the present study provided an interactive experience by superimposing digital content directly on real space on the mobile device after recognizing a marker image via the camera’s lens, the maximum visual range of the user was considered. In the UI’s design, in addition to considering top-to-bottom and left-to-right reading habits, emphasis was placed on ensuring that the user would still be able to easily tap and switch with his/her thumbs when holding the device with both hands. Therefore, the most frequently used main function menu was placed in the lower left part of the UI, and the title function was fixed in the upper left part of the UI. Through simple visual focusing, users were guided as they completed the main task (Fig. 3). All the interactive functions were included in the main function menu, and all the experience functions were switched by tapping.

Fig. 3.
figure 3

Main function menu

Full size image

4 Validation of the Developed AR Application Design

The present study used the mobile-specific heuristic guidelines proposed by Gómez et al. [10] for mobile UIs as a usability checklist for the MAR application that was designed. The 10 usability heuristics proposed by Nielsen were used as the heuristic evaluation (HE) method. HE not only has an extensive coverage of and a high explanatory power for usability but is also efficient, accurate and low-cost. Therefore, HE is widely used to measure the usability of system UIs. Here, the usability of the UIs of mobile devices is considered as an example. With the development of new technology and updated devices, Gómez et al. [10] noted that it is necessary to develop heuristic guidelines for mobile environments. Therefore, they adapted traditional usability heuristics, rearranged them into a new heuristic classification and developed a compilation of heuristic evaluation checklists for mobile interfaces. Then, the resulting mobile-specific HE checklist was used to verify the usability of the UI of the MAR application developed in the present study.

In the present study, three experts in the field of digital media design and three experts in the field of information engineering were invited to measure the UI’s usability. These six experts all had knowledge and experience relating to system development and UI usability and could effectively identify potential problems in the usability of a system’s UI. In addition, some subheuristics from the mobile-specific HE checklist, such as searching, inputting data, shopping, banking and privacy were not included in the functions of the MAR application designed in the present study. Therefore, two subheuristics were moved from the original 13 heuristics. The results of the evaluation based on 11 usability heuristics were as follows:

  • Visibility of the system’s status: Feedback and the system’s status were provided in a timely fashion, and the UI provided sufficient clues to help users operate it.

  • Match between the system and the real world (mental model accuracy): Proper analogies and content conforming to the mental models of users were used to help users fully understand the UI’s functions, menus and operational guidelines.

  • User control and freedom: Straightforward gestures provided users with high operability during the interaction. The permanent main function menu allowed users to switch between front and back functions at any time.

  • Consistency and standards: The overall design of the UI was highly consistent. Visual displays, such as icons, could also convey the meanings of their functions.

  • Error prevention: The occurrence rate of potential errors was minimized through simple and visual operational procedures.

  • Recognition rather than recall: The title of each function page was displayed in the form of an icon complemented by text, which efficiently improved the level of recognition of the system and reduced the knowledge burden on the user.

  • Flexibility and efficiency of use: Interactive switching via the main function menu complemented by image icons provided users with a visual and fast interactive interface design.

  • Aesthetic and minimalist design: The simple and easy-to-understand visual design of the interface allowed users to have a clear and highly immersive interactive experience in the interactive AR environment.

  • Help users recognize, diagnose and recover from errors: The instruction page provided allowed users operating the system for the first time to more rapidly understand its operation. However, there were no auxiliary illustration functions such as interaction cues on the actual interactive pages. Therefore, the experts recommended that auxiliary illustration functions be added to the design of the functional interactive interface.

  • Skills:  The overall operation of the system was quite intuitive and within the range of general users’ abilities.

  • Pleasurable and respectful interaction: Operations could be performed using gestures, giving users a higher degree of freedom of control. In addition, system feedback with the user as the center could improve the level of satisfaction with interactions.

5 Conclusion

With the rapid popularization of mobile carriers and the increasing maturation of AR technology, MAR applications can create more realistic and personalized interactive experiences and marketing services for consumers. MAR services allow a user to interact with virtual information in the real word through the UI of his/her mobile device. However, it is necessary to introduce AR-related usability principles when developing and designing MAR services to ensure that they conform to the usability principles and provide user experiences that meet users’ expectations. On this basis, the present study designed an MAR application aimed at product marketing and providing an interactive experience based on Ko et al.’s MAR usability principles. In addition, experts were invited to measure the usability of the system based on Gómez et al.’s mobile-specific HE checklist. The results demonstrated that the MAR application designed in the present study met the system usability requirements and could provide users with a positive experience during their interactions with it. In the future, we will conduct further usability evaluations involving users and investigate their acceptance of and satisfaction with this MAR application.