Advertisement

Automatic visual pattern mining from categorical image dataset

  • Hongzhi LiEmail author
  • Joseph G. Ellis
  • Lei Zhang
  • Shih-Fu Chang
Regular Paper
  • 28 Downloads

Abstract

We study in this paper the problem of visual pattern mining, which is to identify visually distinctive and semantically meaningful regions in images for solving various visual recognition tasks. Toward this goal, we propose a novel deep neural network architecture called PatternNet for discovering visual patterns that are both discriminative and representative. The proposed PatternNet leverages the filters in the last convolution layer of a convolutional neural network to find locally consistent visual patches, and by combining these filters we can effectively discover unique visual patterns. In addition, PatternNet can discover visual patterns efficiently without performing expensive image patch sampling, and this advantage provides an order of magnitude speedup compared to most other approaches. We evaluate the proposed PatternNet subjectively by showing randomly selected visual patterns which are discovered by our method and quantitatively by performing image classification with the identified visual patterns and comparing our performance with the current state-of-the-art. We also directly evaluate the quality of the discovered visual patterns by leveraging the identified patterns as proposed objects in an image and compare with other relevant methods.

Notes

References

  1. 1.
    Agrawal R, Imieliński T, Swami A (1993) Mining association rules between sets of items in large databases. In: ACM SIGMOD record, vol 22, pp 207–216. ACMGoogle Scholar
  2. 2.
    Alexe B, Deselaers T, Ferrari V (2012) Measuring the objectness of image windows. IEEE Trans Pattern Anal Mach Intell 34(11):2189–2202CrossRefGoogle Scholar
  3. 3.
    Berg T, Belhumeur PN (2013) Poof: part-based one-vs.-one features for fine-grained categorization, face verification, and attribute estimation. In: 2013 IEEE conference on computer vision and pattern recognition (CVPR), pp 955–962. IEEEGoogle Scholar
  4. 4.
    Carreira J, Sminchisescu C (2010) Constrained parametric min-cuts for automatic object segmentation. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR), pp 3241–3248. IEEEGoogle Scholar
  5. 5.
    Chai Y, Lempitsky V, Zisserman A (2013) Symbiotic segmentation and part localization for fine-grained categorization. In: 2013 IEEE international conference on computer vision (ICCV), pp 321–328. IEEEGoogle Scholar
  6. 6.
    Chen G, Yang J, Jin H, Shechtman E, Brandt J, Han TX (2015) Selective pooling vector for fine-grained recognition. In: 2015 IEEE winter conference on applications of computer vision (WACV), pp 860–867. IEEEGoogle Scholar
  7. 7.
    Doersch C, Gupta A, Efros AA (2013) Mid-level visual element discovery as discriminative mode seeking. In: Advances in neural information processing systems, pp 494–502Google Scholar
  8. 8.
    Endres I, Hoiem D (2010) Category independent object proposals. In: Computer vision–ECCV 2010, pp 575–588. SpringerGoogle Scholar
  9. 9.
    Gavves E, Fernando B, Snoek CG, Smeulders AW, Tuytelaars T (2013) Fine-grained categorization by alignments. In: Proceedings of the IEEE international conference on computer vision, pp 1713–1720Google Scholar
  10. 10.
    Gavves E, Fernando B, Snoek CG, Smeulders AW, Tuytelaars T (2014) Local alignments for fine-grained categorization. Int J Comput Vis 111(2):191–212CrossRefGoogle Scholar
  11. 11.
    Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE conference on computer vision and pattern recognition (CVPR), pp 580–587. IEEEGoogle Scholar
  12. 12.
    Hariharan B, Arbeláez P, Girshick R, Malik J (2015) Hypercolumns for object segmentation and fine-grained localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 447–456Google Scholar
  13. 13.
    Harzallah H, Jurie F, Schmid C (2009) Combining efficient object localization and image classification. In: 2009 IEEE 12th international conference on computer vision, pp 237–244. IEEEGoogle Scholar
  14. 14.
    He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: 2017 IEEE international conference on computer vision (ICCV), pp 2980–2988. IEEEGoogle Scholar
  15. 15.
    Juneja M, Vedaldi A, Jawahar C, Zisserman A (2013) Blocks that shout: distinctive parts for scene classification. In: 2013 IEEE conference on computer vision and pattern recognition (CVPR), pp 923–930. IEEEGoogle Scholar
  16. 16.
    Krause J, Jin H, Yang J, Fei-Fei L (2015) Fine-grained recognition without part annotations. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5546–5555Google Scholar
  17. 17.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
  18. 18.
    Li LJ, Su H, Fei-Fei L, Xing EP (2010) Object bank: a high-level image representation for scene classification & semantic feature sparsification. In: Advances in neural information processing systems, pp 1378–1386Google Scholar
  19. 19.
    Li Q, Wu J, Tu Z (2013) Harvesting mid-level visual concepts from large-scale internet images. In: 2013 IEEE conference on computer vision and pattern recognition (CVPR), pp 851–858. IEEEGoogle Scholar
  20. 20.
    Li Y, Liu L, Shen C, van den Hengel A (2015) Mid-level deep pattern mining. In: CVPR, pp 971–980Google Scholar
  21. 21.
    Lowe DG (1999) Object recognition from local scale-invariant features. In: The proceedings of the seventh IEEE international conference on computer vision, 1999, vol 2, pp 1150–1157. IEEEGoogle Scholar
  22. 22.
    Parizi SN, Vedaldi A, Zisserman A, Felzenszwalb P (2014) Automatic discovery and optimization of parts for image classification. arXiv preprint arXiv:1412.6598
  23. 23.
    Pu J, Jiang YG, Wang J, Xue X (2014) Which looks like which: exploring inter-class relationships in fine-grained visual categorization. In: Computer vision–ECCV 2014, pp 425–440. SpringerGoogle Scholar
  24. 24.
    Quattoni A, Torralba A (2009) Recognizing indoor scenes. In: IEEE conference on computer vision and pattern recognition, CVPR 2009, pp 413–420. IEEEGoogle Scholar
  25. 25.
    Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788Google Scholar
  26. 26.
    Sermanet P, Frome A, Real E (2014) Attention for fine-grained categorization. arXiv preprint arXiv:1412.7054
  27. 27.
    Shou Z, Gao H, Zhang L, Miyazawa K, Chang SF (2018) Autoloc: weakly supervised temporal action localization in untrimmed videos. In: ECCV, pp 162–179Google Scholar
  28. 28.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
  29. 29.
    Singh S, Gupta A, Efros A (2012) Unsupervised discovery of mid-level discriminative patches. In: Computer vision-ECCV 2012, pp 73–86Google Scholar
  30. 30.
    Sun J, Ponce J (2013) Learning discriminative part detectors for image classification and cosegmentation. In: 2013 IEEE international conference on computer vision (ICCV), pp 3400–3407. IEEEGoogle Scholar
  31. 31.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9Google Scholar
  32. 32.
    Uijlings JR, van de Sande KE, Gevers T, Smeulders AW (2013) Selective search for object recognition. Int J Comput Vis 104(2):154–171CrossRefGoogle Scholar
  33. 33.
    Vedaldi A, Gulshan V, Varma M, Zisserman A (2009) Multiple kernels for object detection. In: 2009 IEEE 12th international conference on computer vision, pp 606–613. IEEEGoogle Scholar
  34. 34.
    Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: Computer vision–ECCV 2014, pp 818–833. SpringerGoogle Scholar
  35. 35.
    Zhang N, Donahue J, Girshick R, Darrell T (2014) Part-based R-CNNs for fine-grained category detection. In: Computer vision–ECCV 2014, pp 834–849. SpringerGoogle Scholar
  36. 36.
    Zhang W, Li H, Ngo CW, Chang SF (2014) Scalable visual instance mining with threads of features. In: Proceedings of the ACM international conference on multimedia, pp 297–306. ACMGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  • Hongzhi Li
    • 1
    Email author
  • Joseph G. Ellis
    • 2
  • Lei Zhang
    • 1
  • Shih-Fu Chang
    • 2
  1. 1.Microsoft ResearchRedmondUSA
  2. 2.Columbia UniversityNew YorkUSA

Personalised recommendations