Virtual Reality

, Volume 23, Issue 3, pp 281–291 | Cite as

An augmented reality application for improving shopping experience in large retail stores

  • Edmanuel Cruz
  • Sergio Orts-Escolano
  • Francisco Gomez-Donoso
  • Carlos Rizo
  • Jose Carlos Rangel
  • Higinio Mora
  • Miguel CazorlaEmail author
S.I. : Virtual Reality, Augmented Reality and Commerce


In several large retail stores, such as malls, sport or food stores, the customer often feels lost due to the difficulty in finding a product. Although these large stores usually have visual signs to guide customers toward specific products, sometimes these signs are also hard to find and are not updated. In this paper, we propose a system that jointly combines deep learning and augmented reality techniques to provide the customer with useful information. First, the proposed system learns the visual appearance of different areas in the store using a deep learning architecture. Then, customers can use their mobile devices to take a picture of the area where they are located within the store. Uploading this image to the system trained for image classification, we are able to identify the area where the customer is located. Then, using this information and novel augmented reality techniques, we provide information about the area where the customer is located: route to another area where a product is available, 3D product visualization, user location, analytics, etc. The system developed is able to successfully locate a user in an example store with 98% accuracy. The combination of deep learning systems together with augmented reality techniques shows promising results toward improving user experience in retail/commerce applications: branding, advance visualization, personalization, enhanced customer experience, etc.


Smart shopping Deep learning Augmented reality Retail stores User experience Human–computer interaction 3D visualization 



This work has been supported by the Spanish Government TIN2016-76515-R Grant, supported with Feder funds. It has also been supported by the University of Alicante Project GRE16-19.


  1. Ahn J, Williamson J, Gartrell M, Han R, Lv Q, Mishra S (2015) Supporting healthy grocery shopping via mobile augmented reality. ACM Trans Multimedia Comput Commun Appl 12(1s):16:1–16:24. ISSN 1551-6857.
  2. Akgul O, Penekli HI, Genc Y (2016) Applying deep learning in augmented reality tracking. In: 12th international conference on signal-image technology internet-based systems (SITIS), 2016, pp 47–54.
  3. Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B (2001) Recent advances in augmented reality. IEEE Comput Graph Appl 21(6):34–37CrossRefGoogle Scholar
  4. Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. Trans Neural Netw 5(2):157–166CrossRefGoogle Scholar
  5. Blazquez M (2014) Fashion shopping in multichannel retail: the role of technology in enhancing the customer experience. Int J Electron Commer 18(4):97–116CrossRefGoogle Scholar
  6. Carmigniani J, Furht B, Anisetti M, Ceravolo P, Damiani E, Ivkovic M (2011) Augmented reality technologies, systems and applications. Multimed Tools Appl 51(1):341–377CrossRefGoogle Scholar
  7. Chatzopoulos D, Bermejo C, Huang Z, Hui P (2017) Mobile augmented reality survey: from where we are to where we go. IEEE Access 5:6917–6950. ISSN 2169-3536.
  8. Clarifai (2015) Clarifai: Amplifying Intelligence with a Vision. Accessed 1 June 2017
  9. Csurka G, Dance CR, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints. In: Workshop on statistical learning in computer vision, ECCV, pp 1–22Google Scholar
  10. Goldman A (2001) The transfer of retail formats into developing economies: the example of china. J Retail 77(2):221–242. ISSN 0022-4359.
  11. Grewal D, Roggeveen AL, Nordflt J (2017) The future of retailing. J Retail 93(1):1–6. ISSN 0022-4359. (The Future of Retailing)
  12. He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. CoRR. arXiv:1512.03385
  13. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. CoRR. arXiv:1412.6980
  14. Klein G, Murray D (2009) Parallel tracking and mapping on a camera phone. In: Proceedings of the 8th IEEE international symposium on mixed and augmented reality, 2009, ISMAR ’09, Washington, DC, USA. IEEE Computer Society, pp 83–86. ISBN 978-1-4244-5390-0.
  15. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Proceedings of the 25th international conference on neural information processing systems. NIPS'12, vol 1. Curran Associates Inc., USA, pp 1097–1105Google Scholar
  16. Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: Proceedings of the 2006 IEEE computer society conference on computer vision and pattern recognition—CVPR ’06, Washington, DC, USA, vol 2. IEEE Computer Society, pp 2169–2178. ISBN 0-7695-2597-0.
  17. LeCun Y, Kavukcuoglu K, Farabet C (2010) Convolutional networks and applications in vision. In: Proceedings of IEEE international symposium on circuits and systems (ISCAS), 2010. IEEE, pp 253–256Google Scholar
  18. Lemon KN, Verhoef PC (2016) Understanding customer experience throughout the customer journey. J Mark 80(6):69–96CrossRefGoogle Scholar
  19. Li Z, Hoiem D (2016) Learning without forgetting. Springer International Publishing, Cham, pp 614–629. ISBN 978-3-319-46493-0Google Scholar
  20. Likavec S, Osborne F, Cena F (2008) Property-based semantic similarity and relatedness for improving recommendation accuracy and diversity. Int J Semant Web Inf Syst 11(4):1–40CrossRefGoogle Scholar
  21. Maron O, Ratan AL (1998) Multiple-instance learning for natural scene classification. ICML 98:341–349Google Scholar
  22. Martínez-Gómez J, Fernández-Caballero A, García-Varea I, Rodríguez L, Romero-González C (2014) A taxonomy of vision systems for ground mobile robots. Int J Adv Robot Syst 11:1–11CrossRefGoogle Scholar
  23. Martinez-Gomez J, Garcia-Varea I, Cazorla M, Morell V (2015) Vidrilo: the visual and depth robot indoor localization with objects information dataset. Int J Robot Res 34(14):1681–1687. CrossRefGoogle Scholar
  24. Narzt W, Pomberger G, Ferscha A, Kolb D, Müller R, Wieghardt J, Hörtner H, Lindinger C (2006) Augmented reality navigation systems. Univ Access Inf Soc 4(3):177–187CrossRefGoogle Scholar
  25. Newcombe RA, Lovegrove SJ, Davison AJ (2011) DTAM: dense tracking and mapping in real-time. In: Proceedings of the international conference on computer vision, 2011, ICCV ’11, Washington, DC, USA. IEEE Computer Society, pp 2320–2327. ISBN 978-1-4577-1101-5.
  26. Ortiz-Catalan M, Gumundsdttir RA, Kristoffersen MB, Zepeda-Echavarria A, Caine-Winterberger K, Kulbacka-Ortiz K, Widehammar C, Eriksson K, Stockselius A, Ragn C, Pihlar Z, Burger H, Hermansson L (2016) Phantom motor execution facilitated by machine learning and augmented reality as treatment for phantom limb pain: a single group, clinical trial in patients with chronic intractable phantom limb pain. Lancet 388(10062):2885–2894. ISSN 0140-6736Google Scholar
  27. Pascanu R, Mikolov T, Bengio Y (2012) On the difficulty of training recurrent neural networks. ArXiv e-printsGoogle Scholar
  28. Pauly O, Diotte B, Fallavollita P, Weidert S, Euler E, Navab N (2015) Machine learning-based augmented reality for improved surgical scene understanding. Computerized Medical Imaging and Graphics 41(Supplement C):55–60. ISSN 0895-6111. (Machine Learning in Medical Imaging)Google Scholar
  29. Rangel JC, Cazorla M, Garcia-Varea I, Martinez-Gomez J, Fromont E, Sebban M (2016) Scene classification from semantic labeling. Adv Robot 30(11–12):758–769. CrossRefGoogle Scholar
  30. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 1–42. ISSN 0920-5691.
  31. Veijalainen J (2008) Mobile ontologies: concept, development, usage, and business potential. Int J Semant Web Inf Syst 4:20–34CrossRefGoogle Scholar
  32. Willems K, Smolders A, Brengman M, Luyten K, Schning J (2017) The path-to-purchase is paved with digital opportunities: an inventory of shopper-oriented retail technologies. Technol Forecast Soc Change 124(Supplement C):228–242. ISSN 0040-1625Google Scholar
  33. Wu J, Christensen H, Rehg JM et al (2009) Visual place categorization: problem, dataset, and algorithm. In: IEEE/RSJ international conference on intelligent robots and systems, 2009. IROS 2009. IEEE, pp 4763–4770Google Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  1. 1.Instituto Universitario de Investigación InformáticaUniversidad de AlicanteAlicanteSpain

Personalised recommendations