Current State of Mixed Reality Technology for Digital Retail: A Literature Review

Abstract

Immersive 3D environments have been a major research area in different scientific domains such as Human-Computer Interaction, display devices, etc. Mixed Reality (MR) technologies are one of the most interesting sections of immersive environments that have a huge potential of deployment in a diverse range of industries, Retail being one of them. Retail started from traditional physical in-store setups and has been evolving ever since. Integration of technologies gave birth to Digital and Omnichannel Retail and for efficient development and deployment of MR in Omnichannel Retail, appropriate research must be made to provide proper frameworks and guidelines to developers and managers to optimize User Experiences. This paper investigates the current state of the art of MR in Retail by reviewing the present literature in the domain. It proposes important research gaps based on the analysis and understanding to provide further researches a clear picture of the different aspects associated with optimal application of MR technology in Retail sector.

Keywords

1 Introduction

Mixed Reality technology has been expanding ever since it was defined in 1994 [1]. Different industries and firms have been trying to deploy this technology in their respective sectors as it is projected as the future technology by different authors in print and picture media. Despite its existence for several decades, there are gaps and voids that need to be filled for reliable and efficient use of Mixed Reality environments. In the following paper, the current state of the art is discussed by reviewing the existing literature for Mixed Reality and immersive media; the research and development that has been going on around it and the dynamics of its deployment in Retail environments to enhance product experience. From the definition of ‘digitization’ of Retail mentioned in [75] and the trend of Digital Retail mentioned in [3], Digital Retail can be understood as Retail built around or using different digital technologies. It started with Online Retail stores and eventually is moving towards Omnichannel Retailing which integrates traditional physical retailing and Digital Retail in different forms. The paper focuses on analyzing three separate domains: Mixed Reality, Digital Retail and User Experience and tries to blend them to conclude the current state of the art and project future researches on the topic. The aim is to analyze the present literature in Mixed Reality technology in the context of Digital and Omnichannel Retail to identify gaps and propose a future research framework based on these gaps, which could facilitate research and development in the field.

Despite existence of several definitions of Mixed Reality by different authors e.g. [55, 63], we follow the original definition which describes Mixed Reality environment as “one in which real world and virtual world objects are presented together within a single display, that is, anywhere between the extrema of the Virtuality continuum” [1]. The author proposed a whole continuum of immersive environments known as the Virtuality continuum shown in Fig. 1 which encompasses the display devices ranging from physical reality to fully immersive computer-generated environments.

Fig. 1.

Virtuality continuum by Milgram et al. [1]

As described by Microsoft [62], domains like perception and HCI (Human Computer Interaction) are a crucial part of MR as a science. Display technologies have been studied with regards to quality of experience to evaluate perception and Human-Computer Interaction (discussed in Sect. 3) which makes it important to study User Experience which is also crucial to study Omnichannel Retail because the concept of Omnichannel Retail is based on customer holistic experiences (discussed in Sect. 4).

The paper is divided into six further sections, starting with presenting the research methodology. The article then goes on by reviewing the ongoing research and applications of Mixed Reality technology and devices. Section 4 talks about the current state of Mixed Reality environments in Digital and Omnichannel Retail, and the researches and developments that are related to the scenario. The article furthers down to analysis structure in Sect. 5 and description of research gaps in Sect. 6 followed by conclusion, acknowledgement and the used references.

2 Methodology

In the presented article, the used methods for selection of search results merged a protocol-driven methodology (the search strategy is defined at the beginning of the study) with a snowballing technique (the search strategy emerges as the study unfolds) as mentioned in Greenhalgh and Peacock [72] and Ravasi and Stigliani [75]. We used the Google Scholar database and articles were selected by reading abstracts and titles and evaluating the relevance to the topic and the purpose.

The initial search was made to understand the basics of the Mixed Reality technology reaching out to the first published article using the keyword “Mixed Reality” in Google Scholar. Reaching out to the references of the paper, a total collection of 8 articles was made. Two other definitions of MR were established by reaching out to websites of Microsoft and Intel. Another article reviewing Microsoft HoloLens was added from Google Scholar with the search term “Microsoft HoloLens” and industry applications of MR was studied with websites of IKEA, Magic Leap, BMW, and Volvo. Another article was selected from Google scholar to briefly study Pokémon Go with the keyword “Pokémon Go”. These articles combined to form a collection of 16.

In the next phase, search was made with keywords: “3D” + “quality” + “evaluation”, and along with snowballing technique, we collected a set of another 28 articles. The phrase “Mixed Reality user acceptance” was used and snowballed to include another 4 articles. “Product experience” was searched and 15 articles were included. Another 10 articles were obtained using the keywords:” Mixed Reality” + “Retail” giving a total of 73 articles to define the scope of the paper. The research was more focused on studying proven concepts and standards to develop a state of the art and less on identifying pros and cons of individual studies.

3 Mixed Reality Technology

The authors in [1] defined Mixed Reality as a subset of Virtual Reality (VR) displays and taxonomized it into further six classes of displays. Mobile Augmented Reality (AR) which can be considered as class 4 “video see-through” MR displays, has become a major trend among all the other Mixed Reality environments as it has certain advantages such as availability and ease of use; and some trendy applications like Pokémon Go [65] and IKEA Place [66] helped in popularizing it. Class 3 MR display technology is through optical see-through head-mounted displays such as Microsoft HoloLens which was projected as “the future of Augmented Reality” by Microsoft in 2016 [2]. Some early examples of optical see-through displays are [54, 56,57,58]. HoloLens interactions mostly constitute of the ‘HoloLens two core gestures’, some voice-based interactions, gaze-controlled UI pointers and Cortana. As mentioned in [2], we agree to the fact that despite certain hardware and software limitations and shortcomings of the developer version, “future iterations of the HoloLens could profoundly change how we relate to our computers and even to our environment” because users will be able to relate to the physical environment with more digital information and possible interactions, and will be relating to the computer just in a form of daily-usable eye glasses if the designers overcome the hardware limitations. Contrary to the definition followed, Intel describes VR as an umbrella term for all the three technologies: VR, AR and MR and differentiates between AR and MR in parallel [63] which points out the presence of ambiguity of understandings in the domain.

As MR is an immersive display technology, content for the technology is produced in the form of 3D digital objects (often in the form of point clouds and meshes) blended with real environments to different degrees. Different authors have been trying to put up researches to determine the quality and general characteristics associated with 3D content. Visual quality and user perception of the 2D media has been standardized since long, along with standard objective metrics and subjective experimentation methodologies which are important to facilitate perception of the content [67,68,69,70]. The present literature suggests that the visual quality of 3D content goes beyond the concept of just pixel density or resolution like in 2D media. In [46], the authors propose a model-based perception quality assessment for meshes which makes use of both spatial and temporal features. In [43], the authors experimented with geometric and texture noises of 3D objects subjectively and concluded results such as “human viewers are far more sensitive to the distortion of texture than to that of geometry”. In [44], “based on analysis of the subjective quality results, the authors proposed two new metrics for visual quality assessment of textured mesh, as optimized linear combinations of accurate geometry and texture quality measurements. These proposed perceptual metrics outperform their counterparts in terms of correlation with human opinion”. In [28], authors experimented with visual quality features like data corruption in point clouds by noise, simplification, and compression; and concluded that “when the contents are subject to compression like distortions, the underlying surface and shape of the content seem to play a significant role”. In [31], authors worked on contrast sensitivity and discrimination. In [33], geometric distortions introduced by compressing point clouds have been evaluated by subjective and objective methods. In [34], different configurations of the acquisition methodology have been evaluated with subjective methods for multi-view videos. [35] talks about improving techniques for surface construction in 3D models from point cloud data. [37] evaluated quality for 3D point cloud models and concluded important insights about the correlation between human visual system and factors like resolution change and color change in 3D models along with the result that shape has more impact than the color on the quality of 3D model if they are added the almost same amount noise. In [38], authors worked on quality analysis with different network characteristics for 3D videos. In [48], the authors talk about Just Noticeable Difference models for 3D meshes. [32, 40] and [50] talk about quality levels in free viewpoint videos and 3D videos; [36] talks about light field imaging which is another form of immersive 3D environment. Other mentionable works for 3D media are [39, 45, 47] and [51].

As perception of the 3D content is important in Mixed Reality technology as mentioned in [62], determining the quality and User Experience around the content and how it is perceived by the users becomes important. The study around User Experience include topics like HCI which traditionally talks about understanding and optimizing interactions through efficient computer interfaces. The concept of User Interfaces (UI) extended to become User Experience (UX) as recommended in [10], “the term User Experience to be scoped to products, systems, services, and objects that a person interacts with through a user interface”. UX further created the domain of QoE (quality of experience) where researchers and developers are trying to create a holistic hedonic experience for users considering factors like visualization environments, hardware properties, and psychological human factors along with the quality of the content to be visualized [29, 45]. Quality of experience is defined as “the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the user’s personality and current state” [29]. As per our understanding, a QoE centric system is more than just a point and click interface and aims to optimize all the factors that can affect a user’s perception of the content. Factors that could affect the quality of experience are divided into three broad categories of influence factors (IFs) namely Human influence factors, System influence factors, and Context influence factors in [29]. Authors in [47] modeled QoE factors differently into six different categories while [30] describes visual quality in a 3D environment is influenced by three factors: image quality, comfort, and realism.

4 Digital Retail and Mixed Reality

Traditional physical and online retail has been evolved into Multi-channel due to the synergies between online and offline channels of Retail. Multi-channel Retail has been projected to transform into Omnichannel Retail [60] where the concept of channels will slowly diminish, and focus will be more centered on points of contact. “Omnichannel Retail refers to the integration of retail channels like stores, online, and mobile into a single, seamless customer experience” [60]. Authors in [60], which is a Delphi study for the future of Omnichannel Retail, participants agreed that omnichannel will become the new normal over the next ten years and that “the line between [channels] will blur and by the point where no distinction is made”. The authors pointed out some major key trends, challenges, important technologies and main customer touch points in Omnichannel Retail in the coming decade. The authors also pointed out “omnichannel is all about customer convenience […]. detail of the purchase will become competitive”. In 2009, Bourlakis et al. [64] identified a shift from a product orientation to a consumer experience orientation in Retail.

Consumer or customer experience can be studied as a form of User Experience (UX) in the context of Retail when the primary users are the customers. Authors in [10] also tried to define User Experience in a broad domain with the help of experts from industry and academia. Work done in [22] presents a dialogue on User Experience (UX) with an experimental perspective on product quality. In Retail environments, User Experience also extends to product experience which is defined “as the awareness of the psychological effects elicited by the interaction with a product, including the degree to which all our senses are stimulated, the meanings and values we attach to the product, and the feelings and emotions that are elicited” [11]. Another work where product experience is discussed in [16], where the authors talk about product expressions and clustering of those expressions while the work in [23] presents a product experience framework dependent on aesthetic experience, emotional experience, and experience of meaning. In the scope of the paper, the definition in [11] is followed. A conceptual framework for customer experience enhancement is proposed in [20] shown in Fig. 2, that could help the retailers to identify and work on specific “offer zones” to be focused on while designing customer experience in businesses.

Fig. 2.

Conceptual framework for customer experience enhancement [20]

Customer experience is closely associated with customer satisfaction as more satisfied the customer feels with the environment, better the perception will be. The work done in [12] provides a comparison between different scales to measure customer satisfaction based on six criteria: reliability, convergent and discriminant validity, predictive validity, skewness, face validity and managerial value and argued that a 5-point disconfirmation scales would be the preferred method to measure customer satisfaction in contrast to performance and satisfaction scales. In [18], the author suggests that between 10 and 30 customers can be interviewed for one hour to understand the customer needs in a business in a typical study for customer behavior. One of the important works that talks about immersive environments in Retail is [53], which suggests that both enjoyment and engagement positively influence user satisfaction with the simulated Retail environment which influences the intention to purchase the item involved. It also suggests that engagement in the immersive environment explains perceived enjoyment better.

Customer experience is also associated with the acceptance of the technology in Digital Retail environments as it gives an insight to how does the users perceive the used technology based on different factors. Traditionally, there exists some TAM (Technology acceptance models) [15, 26] that have been used to study user acceptance and User Experience for Mixed Reality systems in Digital Retail. In [8], authors study the acceptance and potential of AR applications in Retail point of sale and concluded positively that AR-users rate the offer of information in the store better than non-users and AR-users agree to have found all the information they needed in the store more strongly. User acceptance of MR has been broadly studied in [9] using the traditional constructs like perceived usefulness (PU), perceived ease of use (PEOU), and the intention to use (ITU) [24] along with added constructs: personal innovativeness (PI), perceived enjoyment (PE). Though the work focusses mainly on Mixed Reality technology in education and the acceptance was tested with Mixed Reality regenerative concept (MRRC) for biomedical students, it gives us strong insights about the acceptance of Mixed Reality and the methodology that could be used for testing the acceptance of the technology. More User Experience, product experience and technology acceptance work can be found in [17, 19, 21, 25, 52] and [27]. The authors in [13] studies the current and future of specifically AR in Retail and argues that customer experiences are more realistic when they are embedded, embodied and extended.

MR immersive environments along with providing a more stimulated environment [53] and adding hedonic value to the Retail sector, can also add utilitarian value. Authors in [14] presented a mobile Augmented Reality application that provides customers with product information to assist them in making buying decisions. Pedro Lopes et al. [7] present a smart phone based Mixed Reality application (SPMRA) and concluded through subjective experiments in Germany and Sri Lanka that 84.1% of SPRMA users agreed to the fact that the application assisted and influenced them in making their buying decisions in stores. Automobile manufacturers such as BMW [4] and Volvo [5] has been using Microsoft HoloLens to visualize 3D holograms of automobiles, models, prototypes without the presence of the actual machinery assisting both the manufacturing and the customer showroom experience.

5 Analysis Structure

The three major concepts that were studied in the context of this paper were Mixed Reality, User Experience and Digital/Omnichannel Retail. The collected articles were given a read and the mentioned domains were tried to be synthesized. The articles that contributed to identification of basic concepts including definitions and trends are as follows:

• Mixed Reality: [1, 4,5,6, 54,55,56,57,58,59, 62, 63, 65, 66, 71]

• User Experience: [10, 22]

• Digital/Omnichannel Retail: [3, 60, 61, 64, 74]

The collected articles were then grouped according to the reoccurring themes and concepts found in the articles. These clusters were made by interpreting the overall themes of articles. We identified that evaluating and optimizing experience plays an important role in both Omnichannel Retail and MR developments. The reoccurring themes and concepts were as follows:

• Customer experience: [7, 11, 12, 16, 17, 19, 20, 25, 53, 73]

• Product experience: [7, 21, 23, 24]

• QoE: [29, 30, 45, 47, 49]

• Influence factors in QoE: [28, 31, 33,34,35,36,37,38,39,40,41, 43, 44, 46, 48]

• Pipeline/Analysis of 3D environments: [50, 51, 67,68,69,70]

• Acceptance of technologies: [9, 15, 24, 26, 27]

• Methodologies to evaluate experience: [8, 9, 12, 18]

• Use cases of MR: [7,8,9, 13, 14, 66]

Now the articles clustered as customer experience and product experience talk about different experience frameworks in Retail. They also talk about different components and constraints attached to the experience factors. Extending these frameworks to specifically MR Retail environments couldn’t be described precisely with the literature.

Apart from these frameworks there are factors and models described in immersive environments. The articles clustered as QoE and ‘Influence factors in QoE’ talk about these constraints that affect the perception of a user in 3D immersive environments. When these factors are identified for MR in Retail specifically, the present knowledge is limited and, the relationship with the major customer experience frameworks couldn’t be described properly.

The other four clusters ‘Pipeline/Analysis of 3D environments’, ‘Acceptance of technologies’, ‘Methodologies to evaluate experience’ and ‘Use cases of MR’ described methodologies, analysis deployments that could help in defining the relationship gaps mentioned above.

There are other articles for e.g., [52] that talk about several constraints to customer experience without proposing a holistic experience. The relationship of these factors to the QoE models and ultimately holistic customer experience models couldn’t be described with the present literature in an explicit manner.

Other actors of a Retail ecosystem can also take advantage of MR environments, but the relationship among these actors in context to experience frameworks can be described with very little knowledge.

Initial research gap localization based on these knowledge gaps is shown in Fig. 3.

Fig. 3.

Initial localization of knowledge gaps.

The proposed framework in Fig. 4 was majorly based on customer experience, quality of experience, the QoE influence factors, and the actors in a retail ecosystem. The relationship is described in Sect. 6 clearly. During the analysis, the articles were mapped on to the model. Several experience frameworks were discussed in the article, but the one in [53] and [73] was chosen as it best describes the relationship between immersive environments and customer experience which is the center node of Omnichannel experience. Though no relevant knowledge was found extending this model to MR retail environments in precision. The QoE model was studied in relation to the influence factors. The relationship between the IFs, the antecedents of customer experience, and the customer experience model couldn’t be described precisely with the present literature. Also, the relationship among the actors in the Retail ecosystem and towards the experience framework could not be defined precisely in MR retail environments towards an omnichannel approach.

Fig. 4.

Research and analysis framework

6 Description of Research Gaps

The proposed research plan starts with an initial analysis of the conceptual model used in [53] and originally proposed in [73] where the important factors that are ultimately leading to increased engagement, enjoyment, satisfaction, and ultimately an increased intention to purchase are hedonic experience, utilitarian experience, and simulation experience. Though mostly, only utilitarian and hedonic values are considered as underlying dimensions of customer experience value [52]. The literature suggests that evaluating and optimizing User Experience is an important aspect of MR Omnichannel environments considering customers as the main user of the environments because the concept of Omnichannel Retail is centered around customer holistic experience. Hence, the main objective of integrating MR technologies in Retail should be enhanced customer experience and as [53] suggests, “in the immersive, 3D environment, experience is more associated with engagement and enjoyment, leading to greater purchase intention”. These works talk about the relationships between the components mentioned in the model. This model has been tested on desktop virtual environments and special glasses equipped with “polarised lenses”. Though the authors conceptualize different relationships in virtual and immersive environments in Retail, it lacks its validation in MR Retail environments. We propose the first gap as the validity of this customer experience framework and the different relationships among the components of the model in an MR Retail environment.

The factors included in defining the simulation experience: colour vividness, graphics vividness, and 3D authenticity can be categorized as Content IFs in the QoE model proposed in [29], as they are characteristics of the 3D content in an immersive environment. ‘Control’ as a constraint in simulation experience can be categorized under Context IFs. The proven effect of these characteristics of 3D environments on the components and considering the definition of QoE in [29], these IFs determine the degree of delight or annoyance when perceived by the users, which in the case of Omnichannel Retail are customers. Hence, we conceptualize a relationship between the model proposed in [73] and the QoE factors proposed in [29]. Other Content IFs, broadly System IFs, Context IFs along with Human IFs need to be studied for their impact on hedonic experience, utilitarian experience, engagement, etc. to define this relationship. These proposed studies on the IFs collectively is identified as the second major gap in the present literature. For simplification, this relationship is denoted by an arrow between the two models in the proposed research framework in Fig. 4.

As there exists many IFs that affects the quality and perception of an immersive MR environment, the present literature points out a huge gap of a specific MR customer/User Experience framework in Retail which could point out and taxonomize the MR IFs specifically in Retail, which is the third identified gap.

Some of these IFs are studied in the literature [28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51] for general user perception. Mostly Content IFs are studied widely in these works, but more research is needed in optimizing constraints like physical environment and the social environment of the MR customer experience as mentioned in [52]. The effect of staff characteristics [52] on the perception of MR content is also an open issue. Evaluating the impact of these “Antecedents of customer experience Value” [52] in MR Retail setups can be regarded as the fourth gap in the present literature. As these antecedents can be regarded as influence factors in a Retail environment, they can be categorized or described under the QoE model [29]. Hence, this relationship is represented by an arrow between the QoE model and the model that lists out the antecedents.

We also hypothesize that, as MR environment is a blend of real and digital environments, physical environment can have a huge impact on the user’s perception. The impact of physical environment constraints like the physical hardware constraints to an HMD (Head Mounted Display) mentioned in [45] can be studied to find quantitative correlations that can help further development of MR environments.

These customer experience enhancement frameworks which are based on hedonic and utilitarian value perceived by customers can be extended to other key actors in the sector mentioned in [74] like retailers, suppliers and other intermediaries. The relationship between the experience enhancement framework and different actors is represented by an arrow between the conceptual model and the model that lists out different actors of Retail. The fifth identified gap can be considered as the correlation of perception between different actors in the Retail eco-system which could eventually lead to a unified general perception framework for Retail or a collection of different frameworks for different actors. Retailers like Volvo have already started using it at the designing phase of the production. This gap is could be studied a bit differently as actors which are not customers, have different intentions of using the digital environments.

Use cases like these can add value to the whole Retail eco-system and give insights in different ways if deployed correctly and accepted by the user as technology acceptance is defined as the way people perceive, accept, and adopt technology use [9]. The definition suggest that acceptance of the technology is closely related to user perception of the technology which in turn is related to the experience of the user as if the user has a bad perception about the technology, the holistic experience will not be good for the user. We encourage researchers to build prototype and use cases based on the user perception framework discussed earlier and test them for user acceptance, and further use the results as proof of concept in the Retail sector.

7 Conclusion

In the presented article, three domains: Mixed Reality, User Experience and Digital Retail were studied and synthesized together to analyze the current state of the art. It was evident from the readings that the Retail sector is moving towards Omnichannel Retail model which is centered around customer holistic experience which includes hedonic and utilitarian experience of customers. To provide an efficient hedonic and utilitarian experience, immersive environments in general have proven to be of value. Mixed Reality immersive environments can add value to Omnichannel Retail by getting deployed in Retail ecosystem for different purposes. For efficient deployment of MR into Retail, the relationship between MR, the factors that affect user’s experience of MR and customer experience model in Retail must be defined to facilitate different actors in the Retail ecosystem like developers, managers, customer, etc. This Omnichannel experience model can further be extended to other actors like suppliers, intermediaries, Retailers, etc. to deploy certain use cases for these actors. We identified 5 major gaps based on these relationships between the components of MR technology and Digital Retail that can be acknowledge and studied in the future and help the Retail sector to transition smoothly to customer centric model of Omnichannel approach with qualitative and quantitative studies.