Skip to main content
SpringerLink
Log in

Multitask Learning for Extensive Object Description to Improve Scene Understanding on Monocular Video

  • Conference paper
  • First Online:
Advances in Neural Computation, Machine Learning, and Cognitive Research VI (NEUROINFORMATICS 2022)

Part of the book series: Studies in Computational Intelligence ((SCI,volume 1064))

Included in the following conference series:

  • 517 Accesses

Abstract

Simultaneous localization, classification and association – a pool of the most important tasks in computer vision. With the spread of neural networks to robotic platforms, it has become necessary to develop high-quality and compact machine learning models that allow extensive target object description. In this paper, we supplement the nuScenes dataset with segmentation masks by using the heavy HTC neural network, and also present an algorithm for merging masks with current ground-truth observations. In the end, we are introducing a model that allows us to solve 4 tasks simultaneously using monocular video only: instance segmentation, 2D and 3D rotated box detection, object tracking. The result model runs real-time 18 FPS on NVIDIA RTX2080Ti.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Basharov, I., Yudin, D.: Real-time deep neural networks for multiple object tracking and segmentation on monocular video. Int. Arch. Photogrammetry Remote Sens. Spat. Inf. Sci. 44, 15–20 (2021)

    Article  Google Scholar 

  2. Bergmann, P., Meinhardt, T., Leal-Taixe, L.: Tracking without bells and whistles. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 941–951 (2019)

    Google Scholar 

  3. Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online and realtime tracking. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 3464–3468. IEEE (2016)

    Google Scholar 

  4. Caesar, H., et al.: nuscenes: a multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11621–11631 (2020)

    Google Scholar 

  5. Chen, K., et al.: Hybrid task cascade for instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4974–4983 (2019)

    Google Scholar 

  6. Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., Tian, Q.: Centernet: keypoint triplets for object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6569–6578 (2019)

    Google Scholar 

  7. Jiang, L., Singh, S.S.: Tracking multiple moving objects in images using markov chain monte carlo. Stat. Comput. 28(3), 495–510 (2018). https://doi.org/10.1007/s11222-017-9743-9

    Article  MathSciNet  MATH  Google Scholar 

  8. Kim, A., Ošep, A., Leal-Taixé, L.: EagerMOT: 3D multi-object tracking via sensor fusion. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 11315–11321. IEEE (2021)

    Google Scholar 

  9. Liang, C., Zhang, Z., Zhou, X., Li, B., Lu, Y., Hu, W.: One more check: making “fake background” be tracked again. arXiv preprint arXiv:2104.09441 (2021)

  10. Liang, C., Zhang, Z., Zhou, X., Li, B., Zhu, S., Hu, W.: Rethinking the competition between detection and reid in multiobject tracking. IEEE Trans. Image Process. 31, 3182–3196 (2022)

    Article  Google Scholar 

  11. Luiten, J., Fischer, T., Leibe, B.: Track to reconstruct and reconstruct to track. IEEE Robot. Autom. Lett. 5(2), 1803–1810 (2020)

    Article  Google Scholar 

  12. Voigtlaender, P., et al.: MOTS: multi-object tracking and segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7942–7951 (2019)

    Google Scholar 

  13. Wang, Z., Zheng, L., Liu, Y., Li, Y., Wang, S.: Towards real-time multi-object tracking. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12356, pp. 107–122. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58621-8_7

    Chapter  Google Scholar 

  14. Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 3645–3649. IEEE (2017)

    Google Scholar 

  15. Wu, J., Cao, J., Song, L., Wang, Y., Yang, M., Yuan, J.: Track to detect and segment: an online multi-object tracker. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12352–12361 (2021)

    Google Scholar 

  16. Xu, Y., Osep, A., Ban, Y., Horaud, R., Leal-Taixé, L., Alameda-Pineda, X.: How to train your deep multi-object tracker. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6787–6796 (2020)

    Google Scholar 

  17. Xu, Z., et al.: Segment as points for efficient online multi-object tracking and segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 264–281. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_16

    Chapter  Google Scholar 

  18. Yu, E., Li, Z., Han, S., Wang, H.: RelationTrack: relation-aware multiple object tracking with decoupled representation. In: IEEE Transactions on Multimedia (2022)

    Google Scholar 

  19. Yu, F., Wang, D., Shelhamer, E., Darrell, T.: Deep layer aggregation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2403–2412 (2018)

    Google Scholar 

  20. Zeng, F., Dong, B., Wang, T., Zhang, X., Wei, Y.: Motr: end-to-end multiple-object tracking with transformer. arXiv preprint arXiv:2105.03247 (2021)

  21. Zhang, Y., et al.: ByteTrack: multi-object tracking by associating every detection box. arXiv preprint arXiv:2110.06864 (2021)

  22. Zhang, Y., Wang, C., Wang, X., Zeng, W., Liu, W.: FairMOT: on the fairness of detection and re-identification in multiple object tracking. Int. J. Comput. Vis. 129(11), 3069–3087 (2021). https://doi.org/10.1007/s11263-021-01513-4

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the Russian Science Foundation, project no. 21-71-00131 (https://rscf.ru/project/21-71-00131/).

Author information

Authors and Affiliations

  1. Moscow Institute of Physics and Technology, Intelligent Transport Lab, Moscow, Russia

    Ilya Basharov & Dmitry Yudin

Authors
  1. Ilya Basharov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dmitry Yudin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ilya Basharov .

Editor information

Editors and Affiliations

  1. Scientific Research Institute for System Analysis, Russian Academy of Sciences, Moscow, Russia

    Boris Kryzhanovsky

  2. Scientific Research Institute for System Analysis, Russian Academy of Sciences, Moscow, Russia

    Witali Dunin-Barkowski

  3. Scientific Research Institute for System Analysis, Russian Academy of Sciences, Moscow, Russia

    Vladimir Redko

  4. Moscow Aviation Institute (National Research University), Moscow, Russia

    Yury Tiumentsev

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Basharov, I., Yudin, D. (2023). Multitask Learning for Extensive Object Description to Improve Scene Understanding on Monocular Video. In: Kryzhanovsky, B., Dunin-Barkowski, W., Redko, V., Tiumentsev, Y. (eds) Advances in Neural Computation, Machine Learning, and Cognitive Research VI. NEUROINFORMATICS 2022. Studies in Computational Intelligence, vol 1064. Springer, Cham. https://doi.org/10.1007/978-3-031-19032-2_43

Download citation

Publish with us

Policies and ethics

Search

Navigation