Abstract
Video stitching is a technique that stitches multiple overlapped videos acquired from different cameras, which is widely applied in many applications, including video surveillance, autonomous driving, and virtual reality. Feature-based stitching methods are popular in this area because of their invariance property and efficiency. However, the video stitching pipeline is relatively complicated and the amount of data calculation is large, which impedes its real-time applications. In this paper, we propose a real-time video stitching framework based on vision Digital Signal Processing (DSP). Real-time processing is achieved by the algorithm-level and system-level optimizations. In the algorithm of ORB feature extraction, methods, including look-up table, linear approximation, and single instruction multiple data (SIMD), are adopted to optimize its computation on DSP. In the algorithm of feature matching, the Hamming distance calculation is eliminated for some feature point pairs when their coordinates and angles are not satisfied with certain conditions. Reverse feature matching is proposed to improve registration accuracy. In system-level optimization, the directed acyclic graph (DAG)-based scheduling is proposed to improve the calculation efficiency on dual DSPs, and a ping-pong buffer is utilized to speed up the data transmission between DSP and external memory. Experimental results show that the proposed method can achieve a ten times speedup than that of the CPU, and it can achieve 1536\(\times\)1024@37fps real-time processing on vision DSP.
Similar content being viewed by others
References
Abbadi, N.K.E.L., Al Hassani, S.A., Abdulkhaleq, A.H.: A review over panoramic image stitching techniques. J. Phys.: Conf. Ser. 1999, 0121115 (2021). (IOP Publishing)
Li, C., Liu, J.: Parallax-tolerant image stitching for outdoor scenes. J. Phys.: Conf. Ser. 1651, 012186 (2020). (IOP Publishing)
Guo, C., Jia, F., Tang, W., Huang, P.: A fast method for image matching and registration based on sift algorithm and image pyramid. J. Phys.: Conf. Ser. 1449, 012119 (2020). (IOP Publishing)
Ali, I.H., Salman, S.: 360-degree panoramic image stitching for un-ordered images based on harris corner detection. Indian J. Sci. Technol. 12, 4 (2019)
Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)
Shin, J., Rahim, M.A., Yun, K.S.: Panoramic image stitching with efficient brightness fusion using Ransac algorithm. Int. J. Eng. Technol. 7(3.34), 267–272 (2018)
HajiRassouliha, A., Taberner, A.J., Nash, M.P., Nielsen, P.M.F.: Suitability of recent hardware accelerators (dsps, fpgas, and gpus) for computer vision and image processing algorithms. Signal Process.: Image Commun. 68, 101–119 (2018)
Cadence Design Systems, Inc. Vision P6 DSP User’s Guide, 03 2017
Lee, J.-H.: Panoramic image stitching using feature extracting and matching on embedded system. Trans. Electr. Electron. Mater. 18(5), 273–278 (2017)
Jeon, H., Jeong, J., and Lee, K.: An implementation of the real-time panoramic image stitching using orb and prosac. In: 2015 International SoC Design Conference (ISOCC), pages 91–92. IEEE, (2015)
Qingyi, G., Raut, S., Okumura, K., Aoyama, T., Takaki, T., Ishii, I.: Real-time image mosaicing system using a high-frame-rate video sequence. J. Robot. Mechatron. 27(1), 12–23 (2015)
Jin, H.: Method and apparatus for estimating rotation, focal lengths and radial distortion in panoramic image stitching. US Patent 8, 131–113 (2012)
Hu, K.C., Lin, F.-Y., Chien, C.-C., Tsai, T.-S., Hsia, C.-H., and Chiang, J.-S.. Panoramic image stitching system for automotive applications. In: 2014 IEEE International Conference on Consumer Electronics-Taiwan, pages 203–204. IEEE (2014)
Zhang, H., Zhao, M.: Panoramic image stitching using double encoder-decoders. SN Comput. Sci. 2(2), 1–12 (2021)
Chen, L., Han, J., Zhang, Y., and Bai, L.: Real-time panoramic image mosaic via harris corner detection on fpga. In: Image and Graphics, pages 111–124. Springer (2015)
Tong, L.I.N.G., Xin, Z.H.A.O., Zhe, H.O.U., Kai-wei, W.A.N.G., Jian, B.A.I.: Fast panoramic annular image stretching based on cuda. Comput. Technol. Develop. 27(4), 008–012 (2011)
Torii, A., Dong, Y., Okutomi, M., Sivic, J., Pajdla, T.: Efficient localization of panoramic images using tiled image descriptors. Inf. Media Technol. 9(3), 351–355 (2014)
Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceeding of IEEE International Conference on Computer Vision (1999)
Bay, H., Tuytelaars, T., and Van Gool, L.: Surf: speeded up robust features. In European Conference on Computer Vision, pages 404–417. Springer (2006)
Rublee, E., Rabaud, V., Konolige, K., and Bradski, G.: Orb: an efficient alternative to sift or surf. In 2011 International Conference on Computer Vision, pages 2564–2571. IEEE (2011)
Minchen, Z., Weizhi, W., Binghan, L., Jingshan, H., Derek, A.: Efficient video panoramic image stitching based on an improved selection of Harris corners and a multiple-constraint corner matching. Plos One 8(12), e81182 (2013)
Wang, X., Cao, W., Yao, C., and Yin, H.: Feature matching algorithm based on surf and lowes algorithm. In: 2020 39th Chinese Control Conference (CCC) (2020)
Jae Chang Kwak: An implementation of the real-time image stitching algorithm based on roi. J. IKEEE 19(4), 460–464 (2015)
Bhat, A. S., Shivaprakash, A. V., Prasad, N. S., and Nagaraj, C.: Template matching technique for panoramic image stitching. In: Modelling Symposium (AMS), 2013 7th Asia (2013)
Lo, I. C., Shih, K. T., Yu, P. C.,Hung, C. T. , and Chen, H. H. : Seamless stitching dual fisheye images for 360\(^\circ\) free view. In 2019 IEEE International Conference on Image Processing (ICIP) (2019)
Lee, K. Y. and Sim, J. Y.: Stitching for multi-view videos with large parallax based on adaptive pixel warping. IEEE Access, pp. 1–1 (2018)
Chengyao, D., Yuan, J., Dong, J., Li, L., Chen, M., Li, T.: Gpu based parallel optimization for real time panoramic video stitching. Pattern Recogn. Lett. 133, 62–69 (2020)
Zhi, X., Yan, J., Hang, Y., Wang, S.: Realization of cuda-based real-time registration and target localization for high-resolution video images. J. Real-Time Image Proc. 16(4), 1025–1036 (2019)
Liao, W.-S., Hsieh, T.-J., and Chang, Y.-L.: Gpu parallel computing of spherical panorama video stitching. In 2012 IEEE 18th International Conference on Parallel and Distributed Systems, pages 890–895. IEEE (2012)
Gong, X., Le, Z.: Research and implementation of multi-object tracking based on vision dsp. J. Real-Time Image Proc. 17(6), 1801–1809 (2020)
Turturici, M., Saponara, S., Fanucci, L., and Franchi, E.: Low-power embedded system for real-time correction of fish-eye automotive cameras. In 2012 Design, Automation & Test in Europe Conference & Exhibition (DATE) 2012
Imsaengsuk, T., and Pumrin, S.: Feature detection and description based on orb algorithm for fpga-based image processing. In 2021 9th International Electrical Engineering Congress (iEECON) (2021)
Rosten, E. and Drummond, T.: Machine learning for high-speed corner detection. In European Conference on Computer Vision, pages 430–443. Springer (2006)
Bpl Rosin. Measuring corner properties. In: Computer Vision and Image Understanding. In Cviu (1999)
Calonder, M., Lepetit, V., Strecha, C., and Fua, P.: Brief: binary robust independent elementary features. In European Conference on Computer Vision, pages 778–792. Springer (2010)
Pappalardo, F., Calonaci, C., Pennisi, M., Mastriani, E., and Motta, S.: Hamfast: Fast hamming distance computation. In 2009 WRI World Congress on Computer Science and Information Engineering, volume 1, pages 569–572 (2009)
Yan, C., Zhang, Y., Xu, J., Feng, D., Zhang, J., Dai, Q., Feng, W.: Efficient parallel framework for hevc motion estimation on many-core processors. IEEE Trans. Circ. Syst. Video Technol. 24(12), 2077–2089 (2014)
Gao, W., Ma, S., Duan, L., Tian, Y., Xing, P., Wang, Y., Wang, S., Jia, H., Huang, T.: Digital retina: a way to make the city brain more efficient by visual coding. IEEE Trans. Circ. Syst. Video Technol. 31(11), 4147–4161 (2021)
Balntas, V., Lenc, K., Vedaldi, A., and Mikolajczyk, K.: Hpatches: a benchmark and evaluation of handcrafted and learned local descriptors. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3852–3861 (2017)
Chai, T., Draxler, R.R.: Root mean square error (rmse) or mean absolute error (mae). Geosci. Model Dev. Discuss. 7(1), 1525–1534 (2014)
Sara, U., Akter, M., Uddin, M.S.: Image quality assessment through fsim, ssim, mse and psnr-a comparative study. J. Comput. Commun. 7(3), 8–18 (2019)
Sharma, S.K., Jain, K., and Suresh, M.: Quantitative evaluation of panorama softwares. In International Conference on Communications and Cyber Physical Engineering 2018, pages 543–561. Springer (2018)
Pavan Chennagiri Madhusudana and Rajiv Soundararajan: Subjective and objective quality assessment of stitched images for virtual reality. IEEE Trans. Image Process. 28(11), 5620–5635 (2019)
Jose, A., Pachath, A., Rajesh, A., Chandhan, P., and Shenil, P.S.: Fpga based novel architecture for real-time video stitching. In 2021 Innovations in Power and Advanced Computing Technologies (i-PACT), pages 1–7. IEEE (2021)
Yeh, S.-H., and Lai, S.-H.: Real-time video stitching. In 2017 IEEE International Conference on Image Processing (ICIP), pages 1482–1486. IEEE (2017)
Wang, G., Zhai, Z., Xu, B., and Cheng, Y.: A parallel method for aerial image stitching using orb feature points. In 2017 IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS), pages 769–773. IEEE (2017)
Choi, K., Jun, K.: Real-time panorama video system using networked multiple cameras. J. Syst. Architect. 64, 110–121 (2016)
Mukherjee, S., Su, G.-M., and Cheng, I.: Adaptive dithering using curved Markov-gaussian noise in the quantized domain for mapping sdr to hdr image. In International Conference on Smart Multimedia, pages 193–203. Springer (2018)
Acknowledgements
This work was supported in part by the National Key R &D Program of China (2021ZD0109802), and by the National Natural Science Foundation of China under Grant Nos. 61901150, 61931008, and 61972123.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Huang, X., Tang, R., Zhou, Y. et al. DSP-based parallel optimization for real-time video stitching. J Real-Time Image Proc 20, 28 (2023). https://doi.org/10.1007/s11554-023-01275-x
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11554-023-01275-x