Abstract
We present the first purely event-based, energy-efficient approach for object detection and categorization using an event camera. Compared to traditional frame-based cameras, choosing event cameras results in high temporal resolution (order of microseconds), low power consumption (few hundred mW) and wide dynamic range (120 dB) as attractive properties. However, event-based object recognition systems are far behind their frame-based counterparts in terms of accuracy. To this end, this paper presents an event-based feature extraction method devised by accumulating local activity across the image frame and then applying principal component analysis (PCA) to the normalized neighborhood region. Subsequently, we propose a backtracking-free k-d tree mechanism for efficient feature matching by taking advantage of the low-dimensionality of the feature representation. Additionally, the proposed k-d tree mechanism allows for feature selection to obtain a lower-dimensional dictionary representation when hardware resources are limited to implement dimensionality reduction. Consequently, the proposed system can be realized on a field-programmable gate array (FPGA) device leading to high performance over resource ratio. The proposed system is tested on real-world event-based datasets for object categorization, showing superior classification performance and relevance to state-of-the-art algorithms. Additionally, we verified the object detection method and real-time FPGA performance in lab settings under non-controlled illumination conditions with limited training data and ground truth annotations.
Supported by Temasek Research Fellowship.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Change history
09 July 2019
In the version of this chapter that was originally published, the funding information given at the bottom of the first page was not correct. This has been updated so that the new version now reads: “Supported by Temasek Research Fellowship.”
Notes
References
Beis, J.S., Lowe, D.G.: Shape indexing using approximate nearest-neighbour search in high-dimensional spaces. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society (1997)
Brandli, C., Berner, R., Yang, M., Liu, S.C., Delbruck, T.: A 240 x 180 130 db 3 \(\upmu \)s latency global shutter spatiotemporal vision sensor. IEEE J. Solid-State Circ. 49(10), 2333–2341 (2014)
Conradt, J., Berner, R., Cook, M., Delbruck, T.: An embedded AER dynamic vision sensor for low-latency pole balancing. In: IEEE International Conference on Computer Vision Workshop, pp. 780–785, September 2009
Delbruck, T., Lang, M.: Robotic goalie with 3 ms reaction time at 4% CPU load using event-based dynamic vision sensor. Frontiers Neurosci. 7, 223 (2013)
Galleguillos, C., Rabinovich, A., Belongie, S.: Object categorization using co-occurrence, location and appearance. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE Computer Society (2008)
Hikawa, H., Kaida, K.: Novel FPGA implementation of hand sign recognition system with som-hebb classifier. IEEE Trans. Circ. Syst. Video Technol. 25(1), 153–166 (2015)
Kueng, B., Mueggler, E., Gallego, G., Scaramuzza, D.: Low-latency visual Odometry using event-based feature tracks. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 16–23, October 2016
Lagorce, X., Orchard, G., Gallupi, F., Shi, B.E., Benosman, R.: HOTS: a hierarchy of event-based time-surfaces for pattern recognition. IEEE Trans. Pattern Anal. Mach. Intell. 1(99), 1 (2016)
Lee, J.H., Delbruck, T., Pfeiffer, M.: Training deep spiking neural networks using backpropagation. Frontiers Neurosci. 10, 508 (2016)
Lenz, G., Ieng, S., Benosman, R.: Event-based dynamic face detection and tracking based on activity. CoRR abs/1803.10106 (2018). http://arxiv.org/abs/1803.10106
Liu, H., Moeys, D.P., Das, G., Neil, D., Liu, S.C., Delbruck, T.: Combined frame-and event-based detection and tracking. In: IEEE International Symposium on Circuits and Systems (ISCAS), pp. 2511–2514, May 2016
Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)
Mousouliotis, P.G., Panayiotou, K.L., Tsardoulias, E.G., Petrou, L.P., Symeonidis, A.L.: Expanding a robot’s life: low power object recognition via FPGA-based DCNN deployment. In: 7th International Conference on Modern Circuits and Systems Technologies (MOCAST), pp. 1–4 (2018)
Muja, M., Lowe, D.G.: Fast approximate nearest neighbors with automatic algorithm configuration. In: VISAPP International Conference on Computer Vision Theory and Applications, pp. 331–340 (2009)
Neil, D., Pfeiffer, M., Liu, S.C.: Phased LSTM: accelerating recurrent network training for long or event-based sequences. In: Neural Information Processing Systems, NIPS 2016, pp. 3889–3897. Curran Associates Inc. (2016)
Ni, Z., Bolopion, A., Agnus, J., Benosman, R., Regnier, S.: Asynchronous event-based visual shape tracking for stable haptic feedback in microrobotics. IEEE Trans. Rob. 28(5), 1081–1089 (2012)
O’Connor, P., Neil, D., Liu, S.C., Delbruck, T., Pfeiffer, M.: Real-time classification and sensor fusion with a spiking deep belief network. Frontiers Neurosci. 7 (2013)
Orchard, G., Meyer, C., Etienne-Cummings, R., Posch, C., Thakor, N., Benosman, R.: HFirst: a temporal approach to object recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(10), 2028–2040 (2015)
Orchard, G., Jayawant, A., Cohen, G.K., Thakor, N.: Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers Neurosci. 9, 437 (2015)
Padala, V., Basu, A., Orchard, G.: A noise filtering algorithm for event-based asynchronous change detection image sensors on truenorth and its implementation on truenorth. Frontiers Neurosci. 12, 118 (2018)
Posch, C., Serrano-Gotarredona, T., Linares-Barranco, B., Delbruck, T.: Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc. IEEE 102(10), 1470–1484 (2014)
Ramesh, B., Thi, N.A.L., Orchard, G., Xiang, C.: Spike context: a neuromorphic descriptor for pattern recognition. In: IEEE Biomedical Circuits and Systems Conference (BioCAS), pp. 1–4, October 2017
Ramesh, B., Jian, N.L.Z., Chen, L., et al.: Soft Comput. 23, 2429 (2019). https://doi.org/10.1007/s00500-017-2939-2
Ramesh, B., Yang, H., Orchard, G.M., Le Thi, N.A., Zhang, S., Xiang, C.: DART: distribution aware retinal transform for event-based cameras. IEEE Trans. Pattern Anal. Mach. Intell. (2019). https://doi.org/10.1109/TPAMI.2019.2919301
Redmon, J., Farhadi, A.: YOLOV3: an incremental improvement. CoRR abs/1804.02767 (2018). http://arxiv.org/abs/1804.02767
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017)
Silpa-Anan, C., Hartley, R.: Optimised KD-trees for fast image descriptor matching. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)
Sironi, A., Brambilla, M., Bourdis, N., Lagorce, X., Benosman, R.: HATS: histograms of averaged time surfaces for robust event-based object classification. CoRR abs/1803.07913 (2018). http://arxiv.org/abs/1803.07913
Vikram, T.N., Tscherepanow, M., Wrede, B.: A saliency map based on sampling an image into random rectangular regions of interest. Pattern Recogn. 45(9), 3114–3124 (2012)
Zhai, X., Bensaali, F., McDonald-Maier, K.: Automatic number plate recognition on FPGA. In: International Conference on Electronics, Circuits, and Systems (ICECS), pp. 325–328. IEEE (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Ramesh, B., Ussa, A., Vedova, L.D., Yang, H., Orchard, G. (2019). PCA-RECT: An Energy-Efficient Object Detection Approach for Event Cameras. In: Carneiro, G., You, S. (eds) Computer Vision – ACCV 2018 Workshops. ACCV 2018. Lecture Notes in Computer Science(), vol 11367. Springer, Cham. https://doi.org/10.1007/978-3-030-21074-8_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-21074-8_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-21073-1
Online ISBN: 978-3-030-21074-8
eBook Packages: Computer ScienceComputer Science (R0)