Skip to main content
SpringerLink
Log in

A novel shape descriptor based on salient keypoints detection for binary image matching and retrieval

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

We introduce a shape descriptor that extracts keypoints from binary images and automatically detects the salient ones among them. The proposed descriptor operates as follows: First, the contours of the image are detected and an image transformation is used to generate background information. Next, pixels of the transformed image that have specific characteristics in their local areas are used to extract keypoints. Afterwards, the most salient keypoints are automatically detected by filtering out redundant and sensitive ones. Finally, a feature vector is calculated for each keypoint by using the distribution of contour points in its local area. The proposed descriptor is evaluated using public datasets of silhouette images, handwritten math expressions, hand-drawn diagram sketches, and noisy scanned logos. Experimental results show that the proposed descriptor compares strongly against state of the art methods, and that it is reliable when applied on challenging images such as fluctuated handwriting and noisy scanned images. Furthermore, we integrate our descriptor in a content-based document image retrieval system using sketch queries as a step for query and candidate occurrence matching, and we show that it leads to a significant boost in retrieval performances.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. https://github.com/hchatbri/bsk

References

  1. Abbasi S, Mokhtarian F, Kittler J (1999) Curvature scale space image in shape similarity retrieval. Multimed Syst 7(6):467–476

    Article  Google Scholar 

  2. Baeza-Yates R, Ribeiro-Neto B (1999) Modern information retrieval, vol 463. ACM Press, New York

    Google Scholar 

  3. Bai X, Latecki LJ (2008) Path similarity skeleton graph matching. IEEE Trans Pattern Anal Mach Intell 30(7):1282–1292

    Article  Google Scholar 

  4. Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: European conference on computer vision (ECCV), pp 404–417

    Chapter  Google Scholar 

  5. Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Mach Intell 24(4):509–522

    Article  Google Scholar 

  6. Bober M (2001) MPEG-7 visual shape descriptors. IEEE Trans Circ Syst Vid Technol 11(6):716–719

    Article  Google Scholar 

  7. Breuß M (2013) Innovations for shape analysis: models and algorithms. Springer Science & Business Media

  8. Bunke H, Riesen K (2012) Towards the unification of structural and statistical pattern recognition. Pattern Recogn Lett 33(7):811–825

    Article  Google Scholar 

  9. Cao T-T, Tang K, Mohamed A, Tan T-S (2010) Parallel banding algorithm to compute exact distance transform with the gpu. In: ACM SIGGRAPH Symposium on interactive 3d graphics and games. ACM, pp 83–90

  10. Chalechale A, Naghdy G, Mertins A (2005) Sketch-based image matching using angular partitioning. In: IEEE Transactions on systems, man and cybernetics, part A: systems and humans

    Article  Google Scholar 

  11. Chatbri H, Kameyama K, Kwan P (2013) Sketch-based image retrieval by size-adaptive and noise-robust feature description. In: International conference on digital image computing: techniques and applications (DICTA). IEEE, pp 1–8

  12. Chatbri H, Davila K, Kameyama K, Zanibbi R (2015) Shape matching using keypoints extracted from both the foreground and the background of binary images. In: International Conference on image processing theory, tools and applications (IPTA). IEEE, pp 205–210

  13. Chatbri H, Kameyama K, Kwan P (2015) A comparative study using contours and skeletons as shape representations for binary image matching. Pattern Recognition Letters

  14. Chatbri H, Kameyama K, Kwan P (2015) Towards a segmentation and recognition-free approach for content-based document image retrieval of handwritten queries. In: Asian Conference on pattern recognition (ACPR). IAPR

  15. da S Torres R, Falcao AX (2007) Contour salience descriptors for effective image retrieval and analysis. Image Vis Comput 25(1):3–13

    Article  Google Scholar 

  16. Demirci FM, van Leuken RH, Veltkamp RC (2008) Indexing through laplacian spectra. Comput Vis Image Underst 110(3):312–325

    Article  Google Scholar 

  17. Donoser M, Riemenschneider H, Bischof H (2010) Efficient partial shape matching of outer contours, 281–292

    Chapter  Google Scholar 

  18. Dubey SR et al. (2016) Multichannel decoded local binary patterns for content-based image retrieval. IEEE Trans Image Process 25(9):4018–4032

    Article  MathSciNet  Google Scholar 

  19. Eitz M, Hays J, Alexa M (2012) How do humans sketch objects? ACM Trans Graph 31(4):44–1

    Google Scholar 

  20. Fernanda AA, Paulo AV, da S Torres MR, Falcão AX (2010) Shape feature extraction and description based on tensor scale. Pattern Recogn 43(1):26–36

    Article  Google Scholar 

  21. Fu J, Wang J, Lu H (2010) Effective logo retrieval with adaptive local feature selection. In: ACM International conference on multimedia (ACM MM). ACM, pp 971–974

  22. Harris C, Stephens M (1988) A combined corner and edge detector. In: Alvey vision conference, pp 147–151

  23. Kopf S, Haenselmann T, Effelsberg W (2005) Enhancing curvature scale space features for robust shape classification. In: International conference on multimedia and expo (ICME). IEEE, p 4–pp

  24. Laiche N, Larabi S, Ladraa F, Khadraoui A (2014) Curve normalization for shape retrieval. Signal Process Image Commun 29(4):556–571

    Article  Google Scholar 

  25. Lee S (2013) Symmetry-driven shape description for image retrieval. Image Vis Comput 31(4):357–363

    Article  Google Scholar 

  26. Liang S, Sun Z (2008) Sketch retrieval and relevance feedback with biased svm classification. Pattern Recogn Lett 29(12):1733–1741

    Article  Google Scholar 

  27. Liang S, Li R-H, Baciu G (2011) A graph modeling and matching method for sketch-based garment panel design. In: International conference on cognitive informatics & cognitive Computing (ICCI CC). IEEE, pp 340–347

  28. Liang S, Luo J, Wenyin L, Wei Y (2015) Sketch matching on topology product graph. IEEE Trans Pattern Anal Mach Intell 37(8):1723–1729

    Article  Google Scholar 

  29. Ling H, Jacobs DW (2007) Shape classification using the inner-distance. IEEE Trans Pattern Anal Mach Intell 29(2):286–299

    Article  Google Scholar 

  30. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis

  31. Meijster A, Roerdink JBTM, Hesselink WH (2000) A general algorithm for computing distance transforms in linear time. In: Mathematical morphology and its applications to image and signal processing. Springer, pp 331–340

  32. Mokhtarian F, Abbasi S, Kittler J (1996) Robust and efficient shape indexing through curvature scale space. In: British machine and vision conference (BMVC), vol 96

  33. Otsu N (1975) A threshold selection method from gray-level histograms. Automatica 11(285-296):23–27

    Google Scholar 

  34. Pedrosa GV, Barcelos CAZ (2010) Anisotropic diffusion for effective shape corner point detection. Pattern Recogn Lett 31(12):1658–1664

    Article  Google Scholar 

  35. Pedrosa GV, Batista MA, Barcelos CAZ (2013) Image feature descriptor based on shape salience points. Neurocomputing

  36. Qi W, Zou C, Yuan Y, Hongbing L, Yan P (2013) Image registration by normalized mapping. Neurocomputing 101:181–189

    Article  Google Scholar 

  37. Qian Y, Yang Y, Song Y-Z, Xiang T, Hospedales T (2015) Sketch-a-net that beats humans arXiv:1501.07873

  38. Ren Z, Yuan J, Meng J, Zhang Z (2013) Robust part-based hand gesture recognition using kinect sensor. IEEE Trans Multimed 15(5):1110–1120

    Article  Google Scholar 

  39. Richards W, Hoffman DD (1985) Codon constraints on closed 2D shapes. Comput Vis Graph Image Process 31(3):265–281

    Article  Google Scholar 

  40. Roman-Rangel E, Marchand-Maillet S (2014) Hoosc128: a more robust local shape descriptor. In: Pattern recognition, volume 8495 of Lecture notes in computer science. Springer, pp 172–181

  41. Roman-Rangel E, Marchand-Maillet S (2015) Shape-based detection of maya hieroglyphs using weighted bag representations. Pattern Recogn 48(4):1161–1173

    Article  Google Scholar 

  42. Rosenfeld A, Pfaltz JL (1966) Sequential operations in digital picture processing. J ACM (JACM) 13(4):471–494

    Article  Google Scholar 

  43. Roy Davies E (2004) Machine vision: theory, algorithms, practicalities. Elsevier

  44. Saha PK, Borgefors G, Sanniti di Baja G (2015) A survey on skeletonization algorithms and their applications. Pattern Recognition Letters

  45. Sebastian T, Klein P, Kimia B (2001) Recognition of shapes by editing shock graphs. In: International conference on computer vision (ICCV), vol 1. IEEE, pp 755–755

  46. Sebe N, Qi T, Loupias E, Lew MS, Huang TS (2003) Evaluation of salient point techniques. Image Vis Comput 21(13):1087–1095

    Article  Google Scholar 

  47. Sezgin TM, Davis R (2007) Scale-space based feature point detection for digital ink. In: ACM SIGGRAPH 2007 courses. ACM, p 36

  48. Shafait F, Keysers D, Breuel TM (2008) Efficient implementation of local adaptive thresholding techniques using integral images. In: Document recognition and retrieval XV

  49. Shu X, Xiao-Jun W (2011) A novel contour descriptor for 2D shape matching and its application to image retrieval. Image Vis Comput 29(4):286–294

    Article  Google Scholar 

  50. Song J, Song Y-Z, Xiang T, Hospedales T, Ruan X (2016) Deep multi-task attribute-driven ranking for fine-grained sketch-based image retrieval. In: British Machine vision conference (BMVC), vol 3

  51. Sundar H, Silver D, Gagvani N, Dickinson S (2003) Skeleton based shape matching and retrieval. In: Shape modeling international. IEEE, pp 130–139

  52. Tyler CW (2002) Human symmetry perception and its computational analysis. Psychology Press

  53. Witkin AP (1984) Scale-space filtering: a new approach to multi-scale description. In: International conference on acoustics, speech, and signal processing (ICASSP), vol 9. IEEE, pp 150–153

  54. Yang M, Kpalma K, Ronsin J (2008) A survey of shape feature extraction techniques. Pattern Recogn, 43–90

  55. Yang X, Liu H, Latecki LJ (2012) Contour-based object detection as dominant set computation. Pattern Recogn 45(5):1927–1936

    Article  Google Scholar 

  56. Zanibbi R, Yu L (2011) Math spotting: retrieving math in technical documents using handwritten query images. In: International Conference on document analysis and recognition (ICDAR)

  57. Zhang D, Guojun L (2004) Review of shape representation and description techniques. Pattern Recogn 37(1):1–19

    Article  Google Scholar 

  58. Zhang D, Lu G (2002) A comparative study of fourier descriptors for shape representation and retrieval. In: Asian Conferernce on computer vision (ACCV), pp 646–651

  59. Zhao Peng, Guoqin W u, Yijuan L u, Xianwen W u, Yao Sheng (2016) A novel hand-drawn sketch descriptor based on the fusion of multiple features. Neurocomputing 213:66–74

    Article  Google Scholar 

  60. Zhu G, Doermann D (2007) Automatic document logo detection. In: International conference on document analysis and recognition (ICDAR), pp 864–868

  61. Zhu F, Xie J, Fang Y (2016) Learning cross-domain neural networks for sketch-based 3d shape retrieval. In: AAAI Conference on artificial intelligence. AAAI Press, pp 3683–3689

Download references

Acknowledgments

This work has emanated from a research grant in part from the Monbukagakusho Scholarship sponsored by the Japanese Government, in part from the Irish Research Council (IRC) under Grant Number GOIPD/2016/61, and in part from Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289 (Insight Centre for Data Analytics). The authors would also like to thank Dr. Richard Zanibbi for providing his dataset [56], Dr. Mathieu Delalandre and Dr. Alireza Alaei for their assistance with extracting logos from the Tobacco 800 dataset, and Dr. Shuang Liang for her assistance with her dataset [28].

Author information

Authors and Affiliations

  1. Insight Centre for Data Analytics, Dublin City University, Dublin, Ireland

    Houssem Chatbri, Suzanne Little & Noel E. O’Connor

  2. Faculty of Engineering, Information and Systems, University of Tsukuba, Tsukuba, Japan

    Keisuke Kameyama

  3. School of Science and Technology, University of New England, Armidale, NSW, Australia

    Paul Kwan

Authors
  1. Houssem Chatbri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Keisuke Kameyama
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul Kwan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Suzanne Little
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Noel E. O’Connor
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Houssem Chatbri.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chatbri, H., Kameyama, K., Kwan, P. et al. A novel shape descriptor based on salient keypoints detection for binary image matching and retrieval. Multimed Tools Appl 77, 28925–28948 (2018). https://doi.org/10.1007/s11042-018-6054-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6054-x

Keywords

Search

Navigation