Skip to main content
SpringerLink
Log in

Unitail: Detecting, Reading, and Matching in Retail Scene

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13667))

Included in the following conference series:

Abstract

To make full use of computer vision technology in stores, it is required to consider the actual needs that fit the characteristics of the retail scene. Pursuing this goal, we introduce the United Retail Datasets (Unitail), a large-scale benchmark of basic visual tasks on products that challenges algorithms for detecting, reading, and matching. With 1.8M quadrilateral-shaped instances annotated, the Unitail offers a detection dataset to align product appearance better. Furthermore, it provides a gallery-style OCR dataset containing 1454 product categories, 30k text regions, and 21k transcriptions to enable robust reading on products and motivate enhanced product matching. Besides benchmarking the datasets using various start-of-the-arts, we customize a new detector for product detection and provide a simple OCR-based matching solution that verifies its effectiveness. The Unitail and its evaluation server is publicly available at https://unitedretail.github.io.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bai, Y., Chen, Y., Yu, W., Wang, L., Zhang, W.: Products-10K: a large-scale product recognition dataset. CoRR abs/2008.10545 (2020). https://arxiv.org/abs/2008.10545

  2. Cai, Y., Wen, L., Zhang, L., Du, D., Wang, W.: Rethinking object detection in retail stores. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 947–954 (2021)

    Google Scholar 

  3. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. CoRR abs/2005.12872 (2020). https://arxiv.org/abs/2005.12872

  4. Chen, F., Zhu, C., Shen, Z., Zhang, H., Savvides, M.: NCMS: towards accurate anchor free object detection through l2 norm calibration and multi-feature selection. Comput. Vis. Image Underst. 200, 103050 (2020)

    Article  Google Scholar 

  5. Cheng, L., et al.: Weakly supervised learning with side information for noisy labeled images. CoRR abs/2008.11586 (2020). https://arxiv.org/abs/2008.11586

  6. Chng, C.K., Chan, C.S.: Total-text: a comprehensive dataset for scene text detection and recognition. CoRR abs/1710.10400 (2017). http://arxiv.org/abs/1710.10400

  7. Collins, J., et al.: ABO: dataset and benchmarks for real-world 3D object understanding. CoRR abs/2110.06199 (2021). https://arxiv.org/abs/2110.06199

  8. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018). http://arxiv.org/abs/1810.04805

  9. Ding, J., et al.: Object detection in aerial images: a large-scale benchmark and challenges (2021)

    Google Scholar 

  10. Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., Tian, Q.: CenterNet: object detection with keypoint triplets. arXiv preprint arXiv:1904.08189 (2019)

  11. Everingham, M., Gool, L.V., Williams, C.K.I., Winn, J.M., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010). https://doi.org/10.1007/s11263-009-0275-4

    Article  Google Scholar 

  12. Fang, S., Xie, H., Wang, Y., Mao, Z., Zhang, Y.: Read like humans: autonomous, bidirectional and iterative language modeling for scene text recognition (2021)

    Google Scholar 

  13. Follmann, P., Böttger, T., Härtinger, P., König, R., Ulrich, M.: MVTec D2S: densely segmented supermarket dataset. CoRR abs/1804.08292 (2018). http://arxiv.org/abs/1804.08292

  14. George, M., Floerkemeier, C.: Recognizing products: a per-exemplar multi-label image classification approach. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8690, pp. 440–455. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10605-2_29

    Chapter  Google Scholar 

  15. George, M., Mircic, D., Sörös, G., Floerkemeier, C., Mattern, F.: Fine-grained product class recognition for assisted shopping (2015). https://doi.org/10.48550/arxiv.1510.04074. https://arxiv.org/abs/1510.04074

  16. Georgiadis, K., et al.: Products-6K: a large-scale groceries product recognition dataset. In: The 14th PErvasive Technologies Related to Assistive Environments Conference, PETRA 2021, pp. 1–7. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3453892.3453894

  17. Goldman, E., Herzig, R., Eisenschtat, A., Goldberger, J., Hassner, T.: Precise detection in densely packed scenes. In: Proceedings of Conference on Computer Vision Pattern Recognition (CVPR) (2019)

    Google Scholar 

  18. Gupta, A., Vedaldi, A., Zisserman, A.: Synthetic data for text localisation in natural images. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)

    Google Scholar 

  19. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)

    Google Scholar 

  20. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  21. Huang, L., Yang, Y., Deng, Y., Yu, Y.: DenseBox: unifying landmark localization with end to end object detection. arXiv preprint arXiv:1509.04874 (2015)

  22. Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Synthetic data and artificial neural networks for natural scene text recognition. In: Workshop on Deep Learning, NIPS (2014)

    Google Scholar 

  23. Jund, P., Abdo, N., Eitel, A., Burgard, W.: The Freiburg groceries dataset. CoRR abs/1611.05799 (2016). http://arxiv.org/abs/1611.05799

  24. Karatzas, D., et al.: ICDAR 2015 competition on robust reading. In: Proceedings of the 2015 13th International Conference on Document Analysis and Recognition (ICDAR), ICDAR 2015, pp. 1156–1160. IEEE Computer Society, New York (2015). https://doi.org/10.1109/ICDAR.2015.7333942

  25. Karatzas, D., et al.: ICDAR 2013 robust reading competition. In: 2013 12th International Conference on Document Analysis and Recognition, pp. 1484–1493 (2013). https://doi.org/10.1109/ICDAR.2013.221

  26. Kong, T., Sun, F., Liu, H., Jiang, Y., Shi, J.: FoveaBox: beyond anchor-based object detector. arXiv preprint arXiv:1904.03797 (2019)

  27. Koubaroulis, D., Matas, J., Kittler, J.: Evaluating colour-based object recognition algorithms using the SOIL-47 database. In: in Asian Conference on Computer Vision, pp. 840–845 (2002)

    Google Scholar 

  28. Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Res. Logist. Q. 2(1–2), 83–97 (1955). https://doi.org/10.1002/nav.3800020109. https://onlinelibrary.wiley.com/doi/abs/10.1002/nav.3800020109

  29. Lee, J., Park, S., Baek, J., Oh, S.J., Kim, S., Lee, H.: On recognizing texts of arbitrary shapes with 2D self-attention. CoRR abs/1910.04396 (2019). http://arxiv.org/abs/1910.04396

  30. Li, H., Wang, P., Shen, C., Zhang, G.: Show, attend and read: a simple and strong baseline for irregular text recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 8610–8617 (2019)

    Google Scholar 

  31. Liao, M., Wan, Z., Yao, C., Chen, K., Bai, X.: Real-time scene text detection with differentiable binarization. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11474–11481 (2020)

    Google Scholar 

  32. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: CVPR (2017)

    Google Scholar 

  33. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  34. Liu, Y., Jin, L., Zhang, S., Zhang, S.: Detecting curve text in the wild: new dataset and new solution. CoRR abs/1712.02170 (2017). http://arxiv.org/abs/1712.02170

  35. Merler, M., Galleguillos, C., Belongie, S.: Recognizing groceries in situ using in vitro training data. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007). https://doi.org/10.1109/CVPR.2007.383486

  36. Ming, Q., Miao, L., Zhou, Z., Yang, X., Dong, Y.: Optimization for arbitrary-oriented object detection via representation invariance loss. IEEE Geosci. Remote Sens. Lett., 1–5 (2021). https://doi.org/10.1109/LGRS.2021.3115110

  37. Mishra, A., Alahari, K., Jawahar, C.: Scene text recognition using higher order language priors. In: BMVC - British Machine Vision Conference. BMVA, Surrey, UK, September 2012. https://doi.org/10.5244/C.26.127. https://hal.inria.fr/hal-00818183

  38. Oucheikh, R., Pettersson, T., Löfström, T.: Product verification using OCR classification and Mondrian conformal prediction. Expert Syste. Appl. 188, 115942 (2022). https://doi.org/10.1016/j.eswa.2021.115942. https://www.sciencedirect.com/science/article/pii/S0957417421012963

  39. Pan, X., Luo, P., Shi, J., Tang, X.: Two at once: enhancing learning and generalization capacities via IBN-Net. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 484–500. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_29

    Chapter  Google Scholar 

  40. Pan, X., et al.: Dynamic refinement network for oriented and densely packed object detection, pp. 1–8 (2020)

    Google Scholar 

  41. Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A.: Image transformer. CoRR abs/1802.05751 (2018). http://arxiv.org/abs/1802.05751

  42. Qian, W., Yang, X., Peng, S., Yan, J., Guo, Y.: Learning modulated loss for rotated object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 3, pp. 2458–2466, May 2021. https://ojs.aaai.org/index.php/AAAI/article/view/16347

  43. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)

    Google Scholar 

  44. Risnumawan, A., Shivakumara, P., Chan, C.S., Tan, C.L.: A robust arbitrary text detection system for natural scene images. Expert Syst. Appl. 41(18), 8027–8048 (2014). https://doi.org/10.1016/j.eswa.2014.07.008. https://www.sciencedirect.com/science/article/pii/S0957417414004060

  45. Rocha, A., Hauagge, D.C., Wainer, J., Goldenstein, S.: Automatic fruit and vegetable classification from images. Comput. Electro. Agric. 70(1), 96–104 (2010). https://doi.org/10.1016/j.compag.2009.09.002. https://www.sciencedirect.com/science/article/pii/S016816990900180X

  46. Rong, T., Zhu, Y., Cai, H., Xiong, Y.: A solution to product detection in densely packed scenes (2021)

    Google Scholar 

  47. Sheng, F., Chen, Z., Xu, B.: NRTR: a no-recurrence sequence-to-sequence model for scene text recognition. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 781–786. IEEE (2019)

    Google Scholar 

  48. Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39, 2298–2304 (2016)

    Article  Google Scholar 

  49. Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., Hassner, T.: TextOCR: towards large-scale end-to-end reasoning for arbitrary-shaped scene text (2021)

    Google Scholar 

  50. Tan, M., Le, Q.V.: EfficientNetV2: smaller models and faster training. CoRR abs/2104.00298 (2021). https://arxiv.org/abs/2104.00298

  51. Tian, Z., Shen, C., Chen, H., He, T.: FCOS: fully convolutional one-stage object detection. arXiv preprint arXiv:1904.01355 (2019)

  52. Varol, G., Kuzu, R.: Toward retail product recognition on grocery shelves. In: International Conference on Graphic and Image Processing (2015)

    Google Scholar 

  53. Wang, W., et al.: Shape robust text detection with progressive scale expansion network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9336–9345 (2019)

    Google Scholar 

  54. Wang, W., et al.: Efficient and accurate arbitrary-shaped text detection with pixel aggregation network. In: ICCV, pp. 8439–8448 (2019)

    Google Scholar 

  55. Wei, X., Cui, Q., Yang, L., Wang, P., Liu, L.: RPC: a large-scale retail product checkout dataset. CoRR abs/1901.07249 (2019). http://arxiv.org/abs/1901.07249

  56. Xu, Y., et al.: Gliding vertex on the horizontal bounding box for multi-oriented object detection. IEEE Trans. Pattern Anal. Mach. Intell. 4, 1452–1459 (2020)

    Google Scholar 

  57. Yang, S., Luo, P., Loy, C.C., Tang, X.: Wider face: a face detection benchmark. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  58. Yang, Z., Liu, S., Hu, H., Wang, L., Lin, S.: RepPoints: point set representation for object detection. arXiv preprint arXiv:1904.11490 (2019)

  59. Yao, C., Bai, X., Liu, W., Ma, Y., Tu, Z.: Detecting texts of arbitrary orientations in natural images. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1083–1090 (2012). https://doi.org/10.1109/CVPR.2012.6247787

  60. Yue, X., Kuang, Z., Lin, C., Sun, H., Zhang, W.: RobustScanner: dynamically enhancing positional clues for robust text recognition. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12364, pp. 135–151. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58529-7_9

    Chapter  Google Scholar 

  61. Zhang, S., Chi, C., Yao, Y., Lei, Z., Li, S.Z.: Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection (2020)

    Google Scholar 

  62. Zhu, C., Chen, F., Shen, Z., Savvides, M.: Soft anchor-point object detection. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 91–107. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_6

    Chapter  Google Scholar 

  63. Zhu, C., He, Y., Savvides, M.: Feature selective anchor-free module for single-shot object detection. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019

    Google Scholar 

  64. Zhu, Y., Chen, J., Liang, L., Kuang, Z., Jin, L., Zhang, W.: Fourier contour embedding for arbitrary-shaped text detection. In: CVPR (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Carnegie Mellon University, Pittsburgh, PA, 15213, USA

    Fangyi Chen, Han Zhang, Jiachen Dou, Shentong Mo, Hao Chen, Uzair Ahmed, Chenchen Zhu & Marios Savvides

  2. University of Pittsburgh, Pittsburgh, PA, 15213, USA

    Zaiwang Li

  3. Tsinghua University, Beijing, 100084, China

    Yongxin Zhang

Authors
  1. Fangyi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Han Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zaiwang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiachen Dou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shentong Mo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yongxin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Uzair Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Chenchen Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Marios Savvides
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fangyi Chen .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 9605 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, F. et al. (2022). Unitail: Detecting, Reading, and Matching in Retail Scene. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13667. Springer, Cham. https://doi.org/10.1007/978-3-031-20071-7_41

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20071-7_41

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20070-0

  • Online ISBN: 978-3-031-20071-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation