Skip to main content
SpringerLink
Log in

Spatial-Temporal Correlation 3D Vehicle Detection and Tracking System with Multiple Surveillance Cameras

基于多视角道路相机的时空关联三维车辆检测与跟踪系统

  • Original Paper
  • Published:
Journal of Shanghai Jiaotong University (Science) Aims and scope Submit manuscript

Abstract

Compared to 3D object detection using a single camera, multiple cameras can overcome some limitations on field-of-view, occlusion, and low detection confidence. This study employs multiple surveillance cameras and develops a cooperative 3D object detection and tracking framework by incorporating temporal and spatial information. The framework consists of a 3D vehicle detection model, cooperatively spatial-temporal relation scheme, and heuristic camera constellation method. Specifically, the proposed cross-camera association scheme combines the geometric relationship between multiple cameras and objects in corresponding detections. The spatial-temporal method is designed to associate vehicles between different points of view at a single timestamp and fulfill vehicle tracking in the time aspect. The proposed framework is evaluated based on a synthetic co-operative dataset and shows high reliability, where the cooperative perception can recall more than 66% of the trajectory instead of 11% for single-point sensing. This could contribute to full-range surveillance for intelligent transportation systems.

摘要

相较于单个相机的三维目标检测, 使用多个不同视角的相机可以克服一些缺陷, 例如感知范围小、视野遮挡以及检测置信度低。本文借助多个道路监控摄像头提出了一种利用时空关联信息的联合三维目标检测与跟踪的框架, 这一框架包含三维车辆检测模型、时空关联算法以及启发式的多相机安装位姿选择方法。具体来讲, 提出的跨相机关联策略结合了多个相机和检测到的多个目标之间的几何关系, 时空关联算法用于在某一时刻下关联不同视角的相机视野下的三维车辆完成目标检测, 并且在不同时刻间关联相同车辆从而实现车辆的跟踪。我们使用仿真生成的数据集评估框架的可行性, 实验结果表明联合检测达到了66%的轨迹召回率, 超过了单视角检测的11%的召回率, 这一框架将有助于推动智能交通系统的全覆盖。

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. CHABOT F, CHAOUCH M, RABARISOA J, et al. Deep MANTA: A coarse-to-fine many-task network for joint 2D and 3D vehicle analysis from monocular image [C]//IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1827–1836.

    Google Scholar 

  2. KUNDU A, LI Y, REHG J M. 3D-RCNN: Instance-level 3D object reconstruction via render-and-compare [C]//UT2018 IEEE /CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3559–3568.

    Chapter  Google Scholar 

  3. YOU Y R, WANG Y, CHAO W L, et al. Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving [EB/OL]. (2020-02-15) [2022-04-10]. https://arxiv.org/abs/1906.06310.

  4. RODDICK T, KENDALL A, CIPOLLA C. Orthographic feature transform for monocular 3D object detection [EB/OL]. (2018-11-20) [2022-04-10]. https://arxiv.org/abs/1811.08188.

  5. BRAZIL G, LIU X M. M3D-RPN: Monocular 3D region proposal network for object detection [C]//IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 9286–9295.

    Chapter  Google Scholar 

  6. ZHOU X Y, WANG D Q, KRÄHENBÜHL P. Objects as points [EB/OL]. (2019-04-25) [2022-04-10]. https://arxiv.org/abs/1904.07850.

  7. JÖRGENSENE, ZACHC, KAHL F. Monocular 3D object detection and box fitting trained end-to-end using intersection-over-union loss [EB/OL]. (2019-06-20) [2022-04-10]. https://arxiv.org/abs/1906.08070.

  8. WENG X S, WANG J R, HELD D, et al. 3D multi-object tracking: A baseline and new evaluation metrics [C]//IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Las Vegas: IEEE, 2020: 10359–10366.

    Google Scholar 

  9. PATIL A, MALLA S, GANG H M, et al. The H3D dataset for full-surround 3D multi-object detection and tracking in crowded urban scenes [C]//International Conference on Robotics and Automation (ICRA). Montreal: IEEE, 2019: 9552–9557.

    Chapter  Google Scholar 

  10. OSEP A, MEHNER W, MATHIAS M, et al. Combined image- and world-space tracking in traffic scenes [C]//IEEE International Conference on Robotics and Automation. Singapore: IEEE, 2017: 1988–1995.

    Google Scholar 

  11. DONG C Q, CHEN X W, DONG H B, et al. Research on intelligent vehicle infrastructure cooperative system based on Zigbee [C]//5th International Conference on Transportation Information and Safety (ICTIS). Liverpool: IEEE, 2019: 1337–1343.

    Google Scholar 

  12. MAI X C, YANG M, WANG C X, et al. Multi-sensor fusion based vehicle detection and tracking method [J]. Journal of Shanghai Jiao Tong University, 2011, 45(7): 1012–1016 (in Chinese).

    Google Scholar 

  13. CHEN Q, TANG S H, YANG Q, et al. Cooper: Co-operative perception for connected autonomous vehicles based on 3D point clouds [C]//IEEE 39th International Conference on Distributed Computing Systems. Dallas: IEEE, 2019: 514–524.

    Google Scholar 

  14. NAPHADE M, TANG Z, CHANG M C, et al. The 2019 AI city challenge [C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 452–460.

    Google Scholar 

  15. LUNA E, SANMIGUEL J C, MARTINEZ J M, et al. Online clustering-based multi-camera vehicle tracking in scenarios with overlapping FOVs [J]. Multimedia Tools and Applications, 2022, 81(5): 7063–7083.

    Article  Google Scholar 

  16. MOUSAVIAN A, ANGUELOV D, FLYNN J, et al. 3D bounding box estimation using deep learning and geometry [C]//IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5632–5640.

    Google Scholar 

  17. LAW H, DENG J. CornerNet: Detecting objects as paired keypoints [J]. International Journal of Computer Vision volume, 2020, 128: 642–656.

    Article  Google Scholar 

  18. ESTER M, KRIEGEL H P, SANDER J, et al. A density-based algorithm for discovering clusters in large spatial databases with noise [C]//2nd International Conference on Knowledge Discovery and Data Mining (KDD-96). Portland: AAAI, 1996: 226–231.

    Google Scholar 

  19. KALMAN R E. A new approach to linear filtering and prediction problems [J]. Journal of Basic Engineering, 1960, 82(1): 35–45.

    Article  MathSciNet  Google Scholar 

  20. WOJKE N, BEWLEY A, PAULUS D. Simple online and realtime tracking with a deep association metric [C]//IEEE International Conference on Image Processing. Beijing: IEEE, 2017: 3645–3649.

    Google Scholar 

  21. HANEL M L, SCHONLIEB C B. Efficient global optimization of non-differentiable, symmetric objectives for multi camera placement [J]. IEEE Sensors Journal, 2022, 22(6): 5278–5287.

    Article  Google Scholar 

  22. GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? The KITTI vision benchmark suite [C]//IEEE Conference on Computer Vision and Pattern Recognition. Providence: IEEE, 2012: 3354–3361.

    Google Scholar 

  23. DOSOVITSKIY A, ROS G, CODEVILLA F, et al. CARLA: An open urban driving simulator [C]//1st Conference on Robot Learning. Mountain View: PMLR, 2017: 1–16.

    Google Scholar 

  24. BERNARDIN K, STIEFELHAGEN R. Evaluating multiple object tracking performance: The CLEAR MOT metrics [J]. EURASIP Journal on Image and Video Processing, 2008, 2008: 246309.

    Article  Google Scholar 

  25. YU F, WANG D Q, SHELHAMER E, et al. Deep layer aggregation [C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2403–2412.

    Google Scholar 

Download references

Funding

Foundation item: the National Natural Science Foundation of China (No. 61873167), and the Automotive Industry Science and Technology Development Foundation of Shanghai (No. 1904)

Author information

Authors and Affiliations

  1. Department of Automation, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

    Weipeng Xue  (薛炜彭), Minghu Wu  (吴明虎) & Lin Wang  (王琳)

Authors
  1. Weipeng Xue  (薛炜彭)
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Minghu Wu  (吴明虎)
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lin Wang  (王琳)
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lin Wang  (王琳).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xue, W., Wu, M. & Wang, L. Spatial-Temporal Correlation 3D Vehicle Detection and Tracking System with Multiple Surveillance Cameras. J. Shanghai Jiaotong Univ. (Sci.) 28, 52–60 (2023). https://doi.org/10.1007/s12204-023-2567-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12204-023-2567-1

Key words

关键词

CLC number

Document code

Search

Navigation