Journal of Real-Time Image Processing

, Volume 14, Issue 3, pp 647–665 | Cite as

Robust feature extraction algorithm suitable for real-time embedded applications

  • Abiel Aguilar-GonzálezEmail author
  • Miguel Arias-Estrada
  • François Berry
Special Issue Paper


Smart cameras integrate processing close to the image sensor, so they can deliver high-level information to a host computer or high-level decision process. One of the most common processing is the visual features extraction since many vision-based use-cases are based on such algorithm. Unfortunately, in most of cases, features detection algorithms are not robust or do not reach real-time processing. Based on these limitations, a feature detection algorithm that is robust enough to deliver robust features under any type of indoor/outdoor scenarios is proposed. This was achieved by applying a non-textured corner filter combined to a subpixel refinement. Furthermore, an FPGA architecture is proposed. This architecture allows compact system design, real-time processing for Full HD images (it can process up to 44 frames/91.238.400 pixels per second for Full HD images), and high efficiency for smart camera implementations (similar hardware resources than previous formulations without subpixel refinement and without non-textured corner filter). For accuracy/robustness, experimental results for several real-world scenes are encouraging and show the feasibility of our algorithmic approach.


Robust feature extraction Smart camera FPGA 3D reconstruction 

Supplementary material

11554_2017_701_MOESM1_ESM.mp4 (52.8 mb)
Supplementary material 1 (mp4 54054 KB)


  1. 1.
    Aguilar-González, A., Arias-Estrada, M.: Dense mapping for monocular-slam. In: Indoor Positioning and Indoor Navigation (IPIN), 2016 International Conference on, IEEE, pp. 1–8 (2016)Google Scholar
  2. 2.
    Aguilar-González, A., Arias-Estrada, M.: Towards a smart camera for monocular slam. In: Proceedings of the 10th International Conference on Distributed Smart Camera, ACM, pp. 128–135 (2016)Google Scholar
  3. 3.
    Aguilar-González, A., Arias-Estrada, M., Pérez-Patricio, M., Camas-Anzueto, J.L.: An FPGA 2d-convolution unit based on the CAPH language. J. Real-Time. Image. Proc. (2015). doi: 10.1007/s11554-015-0535-1 Google Scholar
  4. 4.
    Amaricai, A., Gavriliu, C.E., Boncalo, O.: An fpga sliding window-based architecture harris corner detector. In: 2014 24th International Conference on Field Programmable Logic and Applications (FPL), IEEE, pp. 1–4 (2014)Google Scholar
  5. 5.
    Aydogdu, M.F., Demirci, M.F., Kasnakoglu, C.: Pipelining harris corner detection with a tiny FPGA for a mobile robot. In: 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, pp. 2177–2184 (2013)Google Scholar
  6. 6.
    Birem, M., Berry, F.: Dreamcam: a modular FPGA-based smart camera architecture. J. Syst. Archit. 60(6), 519–527 (2014)CrossRefGoogle Scholar
  7. 7.
    Bourrasset, C., Maggianiy, L., Sérot, J., Berry, F., Pagano, P.: Distributed FPGA-based smart camera architecture for computer vision applications. In: 2013 Seventh International Conference on Distributed Smart Cameras (ICDSC), IEEE, pp. 1–2 (2013)Google Scholar
  8. 8.
    Bravo, I., Baliñas, J., Gardel, A., Lázaro, J.L., Espinosa, F., García, J.: Efficient smart cmos camera based on FPGAs oriented to embedded image processing. Sensors 11(3), 2282–2303 (2011)CrossRefGoogle Scholar
  9. 9.
    Brenot, F., Fillatreau, P., Piat, J.: FPGA based accelerator for visual features detection. In: 2015 IEEE International Workshop of Electronics, Control, Measurement, Signals and their Application to Mechatronics (ECMSM), IEEE, pp. 1–6 (2015)Google Scholar
  10. 10.
    Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 6, 679–698 (1986)CrossRefGoogle Scholar
  11. 11.
    Carey, S.J., Barr, D.R., Dudek, P.: Low power high-performance smart camera system based on scamp vision sensor. J. Syst. Archit. 59(10), 889–899 (2013)CrossRefGoogle Scholar
  12. 12.
    Chao, T.L., Wong, K.H.: An efficient FPGA implementation of the harris corner feature detector. In: 2015 14th IAPR International Conference on Machine Vision Applications (MVA), IEEE, pp. 89–93 (2015)Google Scholar
  13. 13.
    Choi, C., Christensen, H.I.: 3d textureless object detection and tracking: An edge-based approach. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, pp. 3877–3884 (2012)Google Scholar
  14. 14.
    Dinh, T.H., Vu, D.Q., Ngo, V.D., Ngoc, N.P., Truong, V.T.: High throughput fpga architecture for corner detection in traffic images. In: 2014 IEEE Fifth International Conference on Communications and Electronics (ICCE), IEEE, pp. 297–302 (2014)Google Scholar
  15. 15.
    Feichtenhofer, C., Pinz, A.: Spatio-temporal good features to track. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 246–253 (2013)Google Scholar
  16. 16.
    Fularz, M., Kraft, M., Schmidt, A., Kasiński, A.: The architecture of an embedded smart camera for intelligent inspection and surveillance. In: Szewczyk, R., Zieliński, C., Kaliczyńska, M. (eds.) Progress in Automation, Robotics and Measuring Techniques, pp. 43–52. Springer, Berlin (2015)Google Scholar
  17. 17.
    Gauglitz, S., Höllerer, T., Turk, M.: Evaluation of interest point detectors and feature descriptors for visual tracking. Int. J. Comput. Vis. 94(3), 335–360 (2011)CrossRefzbMATHGoogle Scholar
  18. 18.
    Gil, A., Mozos, O.M., Ballesta, M., Reinoso, O.: A comparative evaluation of interest point detectors and local descriptors for visual slam. Mach. Vis. Appl. 21(6), 905–920 (2010)CrossRefGoogle Scholar
  19. 19.
    Han, Y., Chen, P., Meng, T.: Harris corner detection algorithm at sub-pixel level and its application. Adv. Comput. Sci. Res. (2015). doi: 10.2991/iccse-15.2015.23 Google Scholar
  20. 20.
    Haritaoglu, I., Harwood, D., Davis, L.S.: W 4: Real-time surveillance of people and their activities. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 809–830 (2000)CrossRefGoogle Scholar
  21. 21.
    Harris, C., Stephens, M.: A combined corner and edge detector. In: Alvey vision conference, Citeseer, vol. 15, p. 50 (1988)Google Scholar
  22. 22.
    Hasegawa, T., Yamauchi, Y., Ambai, M., Yoshida, Y., Fujiyoshi, H.: Keypoint detection by cascaded fast. In: 2014 IEEE International Conference on Image Processing (ICIP), IEEE, pp. 5676–5680 (2014)Google Scholar
  23. 23.
    Hengstler, S., Prashanth, D., Fong, S., Aghajan, H.: Mesheye: a hybrid-resolution smart camera mote for applications in distributed intelligent surveillance. In: Proceedings of the 6th International Conference on Information Processing in Sensor Networks, ACM, pp. 360–369 (2007)Google Scholar
  24. 24.
    Hsiao, P.Y., Lu, C.L., Fu, L.C.: Multilayered image processing for multiscale harris corner detection in digital realization. IEEE Trans. Ind. Electron. 57(5), 1799–1805 (2010)CrossRefGoogle Scholar
  25. 25.
    Köhler, T., Röchter, F., Lindemann, J.P., Möller, R.: Bio-inspired motion detection in an FPGA-based smart camera module. Bioinspir. Biomim. 4(1), 015008 (2009)CrossRefGoogle Scholar
  26. 26.
    Kraft, M., Schmidt, A., Kasinski, A.J.: High-speed image feature detection using FPGA implementation of fast algorithm. VISAPP 1(8), 174–179 (2008)Google Scholar
  27. 27.
    Li, N., Huo, H., Ym, Zhao, Chen, X., Fang, T.: A spatial clustering method with edge weighting for image segmentation. IEEE Geosci. Remote Sens. Lett. 10(5), 1124–1128 (2013)CrossRefGoogle Scholar
  28. 28.
    Li, Y., Chu, W.: A new non-restoring square root algorithm and its VLSI implementations. In: 1996 IEEE International Conference on Computer Design: VLSI in Computers and Processors, 1996. ICCD’96. Proceedings, IEEE, pp. 538–544 (1996)Google Scholar
  29. 29.
    Liu, J., Wark, T., Martin, S., Corke, P., D’Souza, M.: Distributed object tracking with robot and disjoint camera networks. In: 2011 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), IEEE, pp. 380–383 (2011)Google Scholar
  30. 30.
    Mozos, ÓM., Gil, A., Ballesta, M., Reinoso, O.: Interest point detectors for visual slam. In: Conference of the Spanish Association for Artificial Intelligence, Springer, pp. 170–179 (2007)Google Scholar
  31. 31.
    Norouznezhad, E., Bigdeli, A., Postula, A., Lovell, B.C.: Object tracking on fpga-based smart cameras using local oriented energy and phase features. In: Proceedings of the Fourth ACM/IEEE International Conference on Distributed Smart Cameras, ACM, pp. 33–40 (2010)Google Scholar
  32. 32.
    Olson, T., Brill, F.: Moving object detection and event recognition algorithms for smart cameras. In: Proc. DARPA Image Understanding Workshop 20, 205–208 (1997)Google Scholar
  33. 33.
    Possa, P.R., Mahmoudi, S.A., Harb, N., Valderrama, C., Manneback, P.: A multi-resolution FPGA-based architecture for real-time edge and corner detection. IEEE Trans. Comput. 63(10), 2376–2388 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Qureshi, F., Terzopoulos, D.: Smart camera networks in virtual reality. Proc. IEEE 96(10), 1640–1656 (2008)CrossRefGoogle Scholar
  35. 35.
    Rosten, E., Drummond, T.: Fusing points and lines for high performance tracking. In: Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, IEEE, vol. 2, pp. 1508–1515 (2005)Google Scholar
  36. 36.
    Shi, J., Tomasi, C.: Good features to track. In: 1994 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1994. Proceedings CVPR’94, IEEE, pp. 593–600 (1994)Google Scholar
  37. 37.
    Sroba, L., Ravas, R., Grman, J.: The influence of subpixel corner detection to determine the camera displacement. Proc. Eng. 100, 834–840 (2015)CrossRefGoogle Scholar
  38. 38.
    Stock, C., Mühlmann, U., Chandraker, M.K., Pinz, A.: Subpixel Corner Detection for Tracking Applications Using CMOS Camera Technology. Citeseer (2002)Google Scholar
  39. 39.
    Tanskanen, P., Kolev, K., Meier, L., Camposeco, F., Saurer, O., Pollefeys, M.: Live metric 3d reconstruction on mobile phones. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 65–72 (2013)Google Scholar
  40. 40.
    Torr, P.H., Zisserman, A.: Feature based methods for structure and motion estimation. In: International workshop on vision algorithms, Springer, pp. 278–294 (1999)Google Scholar
  41. 41.
    Ttofis, C., Hadjitheophanous, S., Georghiades, A.S., Theocharides, T.: Edge-directed hardware architecture for real-time disparity map computation. IEEE Trans. Comput. 62(4), 690–704 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Turner, L.R.: Inverse of the Vandermonde matrix with applications. NASA technical note (1966). Patent Number: NASA-TN-D-3547Google Scholar
  43. 43.
    Wu, C., Aghajan, H., Kleihorst, R.: Real-time human posture reconstruction in wireless smart camera networks. In: Proceedings of the 7th international conference on Information processing in sensor networks, IEEE Computer Society, pp. 321–331 (2008)Google Scholar
  44. 44.
    Yang, A.Y., Maji, S., Christoudias, C.M., Darrell, T., Malik, J., Sastry, S.S.: Multiple-view object recognition in smart camera networks. In: Distributed Video Sensor Networks, Springer, pp. 55–68 (2011)Google Scholar
  45. 45.
    Yuan, J.Y.: Numerical methods for generalized least squares problems. J. Comput. Appl. Math. 66(1), 571–584 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  46. 46.
    Zhang, G., Vela, P.A.: Good features to track for visual slam. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1373–1382 (2015)Google Scholar
  47. 47.
    Zhang, H. et al.: Quantitative evaluation of feature extractors for visual slam. In: Fourth Canadian Conference on Computer and Robot Vision, 2007. CRV’07, IEEE, pp. 157–164 (2007)Google Scholar
  48. 48.
    Zhu, Q., Wu, B., Wan, N.: A sub-pixel location method for interest points by means of the harris interest strength. Photogramm. Rec. 22(120), 321–335 (2007)CrossRefGoogle Scholar
  49. 49.
    Zhu, W., Ma, C., Xia, L., Li, X.: A fast and accurate algorithm for chessboard corner detection. In: 2nd International Congress on Image and Signal Processing, 2009. CISP’09, IEEE, pp. 1–5 (2009)Google Scholar
  50. 50.
    Zitnick, C.L., Dollár, P.: Edge boxes: Locating object proposals from edges. In: European Conference on Computer Vision, Springer, pp. 391–405 (2014)Google Scholar
  51. 51.
    Zivkovic, Z.: Wireless smart camera network for real-time human 3d pose reconstruction. Comput. Vis. Image Underst. 114(11), 1215–1222 (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  1. 1.Instituto Nacional de Astrofísica Óptica y Electrónica (INAOE)PueblaMexico
  2. 2.Institut PascalUniversité Clermont AuvergneAubière, Clermont-FerrandFrance

Personalised recommendations