Show Something: Intelligent Shopping Assistant Supporting Quick Scene Understanding and Immersive Preview

  • Hao DouEmail author
  • Zhinan Li
  • Minghao Cai
  • Kelvin Cheng
  • Soh Masuko
  • Jiro Tanaka
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11570)


In this paper, we introduce an intelligent shopping assistant system supported by quick scene understanding and augmented-reality 3D preview. By understanding the scene that users are looking at, and using the detected scene information, our system recommends related products that are not in the current scene and which could potentially interest users. With the help of existing speech recognition techniques, our system extracts users’ voice command and keywords, and provides responses in real-time, which allows users to search and filter specific products just by using voice. After finding the potential target products, our system provides users with an augmented-reality preview experience. It automatically brings products to the suitable space in front of the users by using life-size three-dimensional virtual products and spatial understanding. Users can also use two-hand gestural manipulation to operate the virtual products. Through our system, users can obtain products that are strongly related to the current environment, and intuitively preview the products in the current scene by automatic placement and two-hand manipulation to make shopping decisions.


Immersive shopping Intelligent system Context awareness 


  1. 1.
    Luo, P., Yan, S., Liu, Z.Q., Shen, Z.Y., Yang, S.W., He, Q.: From online behaviors to offline retailing. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2016, pp. 175–184 (2016)Google Scholar
  2. 2. Microsoft Cognitive Services Text Analytics API (v2.0) (2018). Accessed 10 Jan. 2019
  3. 3. Gestures - Mixed Reality (2018). Accessed 9 Jan 2019
  4. 4.
    O’Hara, K., Perry, M.: Shopping anytime anywhere, extended abstracts. In: Proceedings of the ACM CHI 2001 Conference on Human Factors in Computing Systems, pp. 345–346 (2001)Google Scholar
  5. 5.
    Li, L.-J., Li, F.-F.: What, where and who? classifying events by scene and object recognition. In: 2007 IEEE 11th International Conference on Computer Vision, pp. 1–8 (2007)Google Scholar
  6. 6.
    Wang, Y.Y., Yu, D., Ju, Y.C., et al.: An introduction to voice search. IEEE Signal Process. Mag 25(3), 28–38 (2008)CrossRefGoogle Scholar
  7. 7. Describing images - Computer Vision - Azure Cognitive Services (2018). Accessed 13 Jan 2019
  8. 8.
    Microsoft 2018b: MixedRealityToolkit-Unity, 29 March 2018. Accessed 12 Jan 2019
  9. 9. Rakuten Web Service: Ichiba Genre Search API (version:2012-07-23)|API (2019). Accessed 12 Jan 2019
  10. 10. Rakuten Web Service: Ichiba Item Search API (version:2017-07-06)|API (2019). Accessed 12 Jan 2019
  11. 11. Voice design - Mixed Reality (2018). Accessed 15 Jan 2019
  12. 12. Voice input - Mixed Reality (2019). Accessed 17 Jan 2019
  13. 13.
    Chaconas, N., Hollerer, T.: An evaluation of bimanual gestures on the microsoft hololens. In: 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 33–40 (2018)Google Scholar
  14. 14.
    Mobile APP| (2019). Accessed 15 Jan 2019
  15. 15.
    Chen, C.C., Huang, T.C., Park, J.J., et al.: A smart assistant toward product-awareness shopping. Pers. Ubiquit. Comput. 18(2), 339–349 (2014)CrossRefGoogle Scholar
  16. 16.
    Cheng, K., Nakazawa, M., Masuko, S.: MR-Shoppingu: physical interaction with augmented retail products using continuous context awareness. In: Munekata, N., Kunita, I., Hoshino, J. (eds.) ICEC 2017. LNCS, vol. 10507, pp. 452–455. Springer, Cham (2017). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Hao Dou
    • 1
    Email author
  • Zhinan Li
    • 1
  • Minghao Cai
    • 1
  • Kelvin Cheng
    • 2
  • Soh Masuko
    • 2
  • Jiro Tanaka
    • 1
  1. 1.Waseda UniversityKitakyushuJapan
  2. 2.Rakuten Institute of TechnologyRakuten, Inc.TokyoJapan

Personalised recommendations