Skip to main content
SpringerLink
Log in

An augmented reality application for improving shopping experience in large retail stores

  • S.I. : Virtual Reality, Augmented Reality and Commerce
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

In several large retail stores, such as malls, sport or food stores, the customer often feels lost due to the difficulty in finding a product. Although these large stores usually have visual signs to guide customers toward specific products, sometimes these signs are also hard to find and are not updated. In this paper, we propose a system that jointly combines deep learning and augmented reality techniques to provide the customer with useful information. First, the proposed system learns the visual appearance of different areas in the store using a deep learning architecture. Then, customers can use their mobile devices to take a picture of the area where they are located within the store. Uploading this image to the system trained for image classification, we are able to identify the area where the customer is located. Then, using this information and novel augmented reality techniques, we provide information about the area where the customer is located: route to another area where a product is available, 3D product visualization, user location, analytics, etc. The system developed is able to successfully locate a user in an example store with 98% accuracy. The combination of deep learning systems together with augmented reality techniques shows promising results toward improving user experience in retail/commerce applications: branding, advance visualization, personalization, enhanced customer experience, etc.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. https://nrf.com/sites/default/files/Images/Media Center/NRF Retail Return Fraud Final_0.pdf.

  2. https://www.kudan.eu/.

References

  • Ahn J, Williamson J, Gartrell M, Han R, Lv Q, Mishra S (2015) Supporting healthy grocery shopping via mobile augmented reality. ACM Trans Multimedia Comput Commun Appl 12(1s):16:1–16:24. ISSN 1551-6857. https://doi.org/10.1145/2808207

  • Akgul O, Penekli HI, Genc Y (2016) Applying deep learning in augmented reality tracking. In: 12th international conference on signal-image technology internet-based systems (SITIS), 2016, pp 47–54. https://doi.org/10.1109/SITIS.2016.17

  • Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B (2001) Recent advances in augmented reality. IEEE Comput Graph Appl 21(6):34–37

    Article  Google Scholar 

  • Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. Trans Neural Netw 5(2):157–166

    Article  Google Scholar 

  • Blazquez M (2014) Fashion shopping in multichannel retail: the role of technology in enhancing the customer experience. Int J Electron Commer 18(4):97–116

    Article  Google Scholar 

  • Carmigniani J, Furht B, Anisetti M, Ceravolo P, Damiani E, Ivkovic M (2011) Augmented reality technologies, systems and applications. Multimed Tools Appl 51(1):341–377

    Article  Google Scholar 

  • Chatzopoulos D, Bermejo C, Huang Z, Hui P (2017) Mobile augmented reality survey: from where we are to where we go. IEEE Access 5:6917–6950. ISSN 2169-3536. https://doi.org/10.1109/ACCESS.2017.2698164

  • Clarifai (2015) Clarifai: Amplifying Intelligence with a Vision. http://www.clarifai.com/. Accessed 1 June 2017

  • Csurka G, Dance CR, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints. In: Workshop on statistical learning in computer vision, ECCV, pp 1–22

  • Goldman A (2001) The transfer of retail formats into developing economies: the example of china. J Retail 77(2):221–242. ISSN 0022-4359. https://doi.org/10.1016/S0022-4359(01)00044-6. http://www.sciencedirect.com/science/article/pii/S0022435901000446

  • Grewal D, Roggeveen AL, Nordflt J (2017) The future of retailing. J Retail 93(1):1–6. ISSN 0022-4359. https://doi.org/10.1016/j.jretai.2016.12.008. http://www.sciencedirect.com/science/article/pii/S0022435916300872. (The Future of Retailing)

  • He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. CoRR. arXiv:1512.03385

  • Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. CoRR. arXiv:1412.6980

  • Klein G, Murray D (2009) Parallel tracking and mapping on a camera phone. In: Proceedings of the 8th IEEE international symposium on mixed and augmented reality, 2009, ISMAR ’09, Washington, DC, USA. IEEE Computer Society, pp 83–86. ISBN 978-1-4244-5390-0. https://doi.org/10.1109/ISMAR.2009.5336495

  • Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Proceedings of the 25th international conference on neural information processing systems. NIPS'12, vol 1. Curran Associates Inc., USA, pp 1097–1105

  • Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: Proceedings of the 2006 IEEE computer society conference on computer vision and pattern recognition—CVPR ’06, Washington, DC, USA, vol 2. IEEE Computer Society, pp 2169–2178. ISBN 0-7695-2597-0. https://doi.org/10.1109/CVPR.2006.68

  • LeCun Y, Kavukcuoglu K, Farabet C (2010) Convolutional networks and applications in vision. In: Proceedings of IEEE international symposium on circuits and systems (ISCAS), 2010. IEEE, pp 253–256

  • Lemon KN, Verhoef PC (2016) Understanding customer experience throughout the customer journey. J Mark 80(6):69–96

    Article  Google Scholar 

  • Li Z, Hoiem D (2016) Learning without forgetting. Springer International Publishing, Cham, pp 614–629. ISBN 978-3-319-46493-0

  • Likavec S, Osborne F, Cena F (2008) Property-based semantic similarity and relatedness for improving recommendation accuracy and diversity. Int J Semant Web Inf Syst 11(4):1–40

    Article  Google Scholar 

  • Maron O, Ratan AL (1998) Multiple-instance learning for natural scene classification. ICML 98:341–349

    Google Scholar 

  • Martínez-Gómez J, Fernández-Caballero A, García-Varea I, Rodríguez L, Romero-González C (2014) A taxonomy of vision systems for ground mobile robots. Int J Adv Robot Syst 11:1–11

    Article  Google Scholar 

  • Martinez-Gomez J, Garcia-Varea I, Cazorla M, Morell V (2015) Vidrilo: the visual and depth robot indoor localization with objects information dataset. Int J Robot Res 34(14):1681–1687. https://doi.org/10.1177/0278364915596058

    Article  Google Scholar 

  • Narzt W, Pomberger G, Ferscha A, Kolb D, Müller R, Wieghardt J, Hörtner H, Lindinger C (2006) Augmented reality navigation systems. Univ Access Inf Soc 4(3):177–187

    Article  Google Scholar 

  • Newcombe RA, Lovegrove SJ, Davison AJ (2011) DTAM: dense tracking and mapping in real-time. In: Proceedings of the international conference on computer vision, 2011, ICCV ’11, Washington, DC, USA. IEEE Computer Society, pp 2320–2327. ISBN 978-1-4577-1101-5. https://doi.org/10.1109/ICCV.2011.6126513

  • Ortiz-Catalan M, Gumundsdttir RA, Kristoffersen MB, Zepeda-Echavarria A, Caine-Winterberger K, Kulbacka-Ortiz K, Widehammar C, Eriksson K, Stockselius A, Ragn C, Pihlar Z, Burger H, Hermansson L (2016) Phantom motor execution facilitated by machine learning and augmented reality as treatment for phantom limb pain: a single group, clinical trial in patients with chronic intractable phantom limb pain. Lancet 388(10062):2885–2894. ISSN 0140-6736

  • Pascanu R, Mikolov T, Bengio Y (2012) On the difficulty of training recurrent neural networks. ArXiv e-prints

  • Pauly O, Diotte B, Fallavollita P, Weidert S, Euler E, Navab N (2015) Machine learning-based augmented reality for improved surgical scene understanding. Computerized Medical Imaging and Graphics 41(Supplement C):55–60. ISSN 0895-6111. (Machine Learning in Medical Imaging)

  • Rangel JC, Cazorla M, Garcia-Varea I, Martinez-Gomez J, Fromont E, Sebban M (2016) Scene classification from semantic labeling. Adv Robot 30(11–12):758–769. https://doi.org/10.1080/01691864.2016.1164621

    Article  Google Scholar 

  • Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 1–42. ISSN 0920-5691. https://doi.org/10.1007/s11263-015-0816-y

  • Veijalainen J (2008) Mobile ontologies: concept, development, usage, and business potential. Int J Semant Web Inf Syst 4:20–34

    Article  Google Scholar 

  • Willems K, Smolders A, Brengman M, Luyten K, Schning J (2017) The path-to-purchase is paved with digital opportunities: an inventory of shopper-oriented retail technologies. Technol Forecast Soc Change 124(Supplement C):228–242. ISSN 0040-1625

  • Wu J, Christensen H, Rehg JM et al (2009) Visual place categorization: problem, dataset, and algorithm. In: IEEE/RSJ international conference on intelligent robots and systems, 2009. IROS 2009. IEEE, pp 4763–4770

Download references

Acknowledgements

This work has been supported by the Spanish Government TIN2016-76515-R Grant, supported with Feder funds. It has also been supported by the University of Alicante Project GRE16-19.

Author information

Authors and Affiliations

  1. Instituto Universitario de Investigación Informática, Universidad de Alicante, Alicante, Spain

    Edmanuel Cruz, Sergio Orts-Escolano, Francisco Gomez-Donoso, Carlos Rizo, Jose Carlos Rangel, Higinio Mora & Miguel Cazorla

Authors
  1. Edmanuel Cruz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sergio Orts-Escolano
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Francisco Gomez-Donoso
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Carlos Rizo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jose Carlos Rangel
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Higinio Mora
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Miguel Cazorla
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miguel Cazorla.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cruz, E., Orts-Escolano, S., Gomez-Donoso, F. et al. An augmented reality application for improving shopping experience in large retail stores. Virtual Reality 23, 281–291 (2019). https://doi.org/10.1007/s10055-018-0338-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-018-0338-3

Keywords

Search

Navigation