Skip to main content
SpringerLink
Log in

Real-time digital twins end-to-end multi-branch object detection with feature level selection for healthcare

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Object detection is one of the most significant tasks in recent computer vision and healthcare study, which also has been applied in many areas. Although some detection frameworks show good performance for some specific datasets, the ambiguity in feature levels of anchor-free detectors still limits the performance of both fully-supervised and cross-dataset settings. Hence, a digital twins end-to-end multi-branch object detection framework with feature level selection is presented in this work. First, a five-level feature pyramid is adopted with a set of detection heads to construct an anchor-free detection backbone. Then, a learning-based selection strategy is presented to help obtain better feature level selection performance. Experimental results on general object detection datasets show that our framework can achieve 39.2 average precision (AP) on the COCO dataset and 10.2 miss rate (MR) on the CityPersons dataset. Furthermore, experimental results on cross-dataset settings, including Cityscapes, Caltech, SIM 10k, KITTI datasets, have also proved the good generalization ability of our framework. Through the optimized models in digital twins, it is also been applied in a pneumonia detection dataset with 49.3 AP. In addition, a large number of comparisons with state-of-the-art works also verify the detection performance and real-time efficiency of the proposed framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Zhou, X., Liang, W., Li, W., Yan, K., Shimizu, S., Kevin, I., Wang, K.: Hierarchical adversarial attacks against graph neural network based iot network intrusion detection system. IEEE Internet Things J. (2021). https://doi.org/10.1109/JIOT.2021.3130434

    Article  Google Scholar 

  2. Mabrouki, J., Azrour, M., Fattah, G., Dhiba, D., ElHajjaji, S.: Intelligent monitoring system for biogas detection based on the internet of things: Mohammedia, Morocco city landfill case. Big Data Min. Anal. 4(1), 10–17 (2021)

    Article  Google Scholar 

  3. Hu, R., Tang, Z.-R., Song, X., Luo, J., Wu, E.Q., Chang, S.: Ensemble echo network with deep architecture for time-series modeling. Neural Comput. Appl. 33(10), 4997–5010 (2021)

    Article  Google Scholar 

  4. Tang, Z., Chen, Y., Wang, Z., Hu, R., Wu, E.Q.: Non-spike timing-dependent plasticity learning mechanism for memristive neural networks. Appl. Intell. 51(6), 3684–3695 (2021)

    Article  Google Scholar 

  5. Hu, R., Zhou, S., Tang, Z.R., Chang, S., Huang, Q., Liu, Y., Han, W., Wu, E.Q.: Dmman: a two-stage audio-visual fusion framework for sound separation and event localization. Neural Netw. 133, 229–239 (2021)

    Article  Google Scholar 

  6. Liu, L., Chen, X., Petinrin, O.O., Zhang, W., Rahaman, S., Tang, Z.-R., Wong, K.-C.: Machine learning protocols in early cancer detection based on liquid biopsy: a survey. Life 11(7), 638 (2021)

    Article  Google Scholar 

  7. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)

  8. Zhou, X., Liang, W., Kevin, I., Wang, K., Yang, L.T.: Deep correlation mining based on hierarchical hybrid networks for heterogeneous big data recommendations. IEEE Trans. Comput. Soc. Syst. 8(1), 171–178 (2020)

    Article  Google Scholar 

  9. Singh, K.K., Singh, A.: Diagnosis of covid-19 from chest x-ray images using wavelets-based depthwise convolution network. Big Data Min. Anal. 4(2), 84–93 (2021)

    Article  Google Scholar 

  10. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)

  11. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: single shot multibox detector. In: European Conference on Computer Vision, pp. 21–37. Springer (2016)

  12. Liang, W., Hu, Y., Zhou, X., Pan, Y., Kevin, I., Wang, K.: Variational few-shot learning for microservice-oriented intrusion detection in distributed industrial iot. IEEE Trans. Ind. Inform. (2021). https://doi.org/10.1109/TII.2021.3116085

    Article  Google Scholar 

  13. Tang, Z., Sun, Z.H., Wu, E.Q., Wei, C.F., Ming, D., Chen, S.: Mrcg: a mri retrieval system with convolutional and graph neural networks for secure and private iomt. IEEE J. Biomed. Health Inform. (2021). https://doi.org/10.1109/JBHI.2021.3130028

    Article  Google Scholar 

  14. Tian, Z., Shen, C., Chen, H., He, T.: Fcos: fully convolutional one-stage object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 9627–9636 (2019)

  15. Liu, W., Liao, S., Ren, W., Hu, W., Yu, Y.: High-level semantic feature detection: a new perspective for pedestrian detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5187–5196 (2019)

  16. Zhou, X., Xu, X., Liang, W., Zeng, Z., Yan, Z.: Deep-learning-enhanced multitarget detection for end-edge-cloud surveillance in smart iot. IEEE Internet Things J. 8(16), 12588–12596 (2021)

    Article  Google Scholar 

  17. Zhou, X., Yang, X., Ma, J., Kevin, I., Wang, K.: Energy efficient smart routing based on link correlation mining for wireless edge computing in iot. IEEE Internet Things J. (2021). https://doi.org/10.1109/JIOT.2021.3077937

    Article  Google Scholar 

  18. Shen, G., Tang, Z.R., Shen, P., Yu, Y.: Hq-trans: a high-quality screening based image translation framework for unsupervised cross-domain pedestrian detection. In: International Conference on Image and Graphics, pp. 16–27. Springer (2021)

  19. Shen, G., Yu, Y., Tang, Z.-R., Chen, H., Zhou, Z.: Hqa-trans: an end-to-end high-quality-awareness image translation framework for unsupervised cross-domain pedestrian detection. IET Comput. Vis. 16(3), 218–229 (2022)

    Article  Google Scholar 

  20. Zhou, X., Xu, X., Liang, W., Zeng, Z., Shimizu, S., Yang, L.T., Jin, Q.: Intelligent small object detection for digital twin in smart manufacturing with industrial cyber-physical systems. IEEE Trans. Ind. Inf. 18(2), 1377–1386 (2021)

    Article  Google Scholar 

  21. Xu, X., Tian, H., Zhang, X., Qi, L., He, Q., Dou, W.: Discov: distributed Covid-19 detection on x-ray images with edge-cloud collaboration. IEEE Trans. Serv. Comput. (2022). https://doi.org/10.1109/TSC.2022.3142265

    Article  Google Scholar 

  22. Dai, J., Li, Y., He, K., Sun, J.: R-fcn: object detection via region-based fully convolutional networks. In: Advances in Neural Information Processing Systems, pp. 379–387 (2016)

  23. Xu, X., Fang, Z., Zhang, J., He, Q., Yu, D., Qi, L., Dou, W.: Edge content caching with deep spatiotemporal residual network for iov in smart city. ACM Trans. Sens. Netw. (TOSN) 17(3), 1–33 (2021)

    Article  Google Scholar 

  24. Law, H., Deng, J.: Cornernet: detecting objects as paired keypoints. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 734–750 (2018)

  25. Yuan, L., He, Q., Chen, F., Zhang, J., Qi, L., Xu, X., Xiang, Y., Yang, Y.: Csedge: enabling collaborative edge storage for multi-access edge computing based on blockchain. IEEE Trans. Parallel Distrib. Syst. 33(8), 1873–1887 (2021)

    Article  Google Scholar 

  26. Zhu, C., He, Y., Savvides, M.: Feature selective anchor-free module for single-shot object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 840–849 (2019)

  27. Qi, L., He, Q., Chen, F., Zhang, X., Dou, W., Ni, Q.: Data-driven web apis recommendation for building web applications. IEEE Trans. Big Data (2020). https://doi.org/10.1109/TBDATA.2020.2975587

    Article  Google Scholar 

  28. Xu, X., Fang, Z., Qi, L., Zhang, X., He, Q., Zhou, X.: Tripres: traffic flow prediction driven resource reservation for multimedia iov with edge computing. ACM Trans. Multimed. Comput. Commun. Appl. (TOMM) 17(2), 1–21 (2021)

    Article  Google Scholar 

  29. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)

  30. Qi, L., Yang, Y., Zhou, X., Rafique, W., Ma, J.: Fast anomaly identification based on multi-aspect data streams for intelligent intrusion detection toward secure industry 4.0. IEEE Trans. Ind. Inform. (2021). https://doi.org/10.1109/TII.2021.3139363

    Article  Google Scholar 

  31. Qi, L., Song, H., Zhang, X., Srivastava, G., Xu, X., Yu, S.: Compatibility-aware web api recommendation for mashup creation via textual description mining. ACM Trans. Multimed. Comput. Commun. Appl. 17(1s), 1–19 (2021)

    Article  Google Scholar 

  32. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)

  33. Shi, J., Wu, J., Anisetti, M., Damiani, E., Jeon, G.: An interval type-2 fuzzy active contour model for auroral oval segmentation. Soft. Comput. 21(9), 2325–2345 (2017)

    Article  Google Scholar 

  34. Jeon, G., Anisetti, M., Wang, L., Damiani, E.: Locally estimated heterogeneity property and its fuzzy filter application for deinterlacing. Inf. Sci. 354, 112–130 (2016)

    Article  Google Scholar 

  35. Qi, L., Hu, C., Zhang, X., Khosravi, M.R., Sharma, S., Pang, S., Wang, T.: Privacy-aware data fusion and prediction with spatial-temporal context for smart city industrial environment. IEEE Trans. Ind. Inf. 17(6), 4159–4167 (2020)

    Article  Google Scholar 

  36. Lin, T.Y., Maire, M.., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: European Conference on Computer Vision, pp. 740–755. Springer (2014)

  37. Zhang, S., Benenson, R., Schiele, B.: Citypersons: a diverse dataset for pedestrian detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3221 (2017)

  38. Chen, Y., Zhao, F., Lu, Y., Chen, X.: Dynamic task offloading for mobile edge computing with hybrid energy supply. Tsinghua Science and Technology, vol. 10 (2021)

  39. Wu, J., Anisetti, M., Wu, W., Damiani, E., Jeon, G.: Bayer demosaicking with polynomial interpolation. IEEE Trans. Image Process. 25(11), 5369–5382 (2016)

    Article  MathSciNet  Google Scholar 

  40. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3223 (2016)

  41. Ahmed, I., Jeon, G.: A real-time person tracking system based on siammask network for intelligent video surveillance. J. Real-Time Image Proc. 18(5), 1803–1814 (2021)

    Article  Google Scholar 

  42. Johnson-Roberson, M., Barto, C., Mehta, R., Sridhar, S.N., Rosaen, K., Vasudevan, R.: Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? (2016). arXiv:1610.01983

  43. Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the kitti dataset. Int. J. Robot. Res. 32(11), 1231–1237 (2013)

    Article  Google Scholar 

  44. Dollár, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: a benchmark. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 304–311. IEEE (2009)

  45. Ahmed, I., Ahmad, M., Jeon, G.: A real-time efficient object segmentation system based on u-net using aerial drone images. J. Real-Time Image Proc. 18(5), 1745–1758 (2021)

    Article  Google Scholar 

  46. Wang, X., Peng, Y., Lu, Y., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2097–2106 (2017)

  47. Xu, X., Li, H., Xu, W., Liu, Z., Yao, L., Dai, F.: Artificial intelligence for edge service optimization in internet of vehicles: a survey. Tsinghua Sci. Technol. 27(2), 270–287 (2021)

    Article  Google Scholar 

  48. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  49. Liu, W., Liao, S., Hu, W., Liang, X., Chen, X.: Learning efficient single-stage pedestrian detectors by asymptotic localization fitting. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 618–634 (2018)

  50. Zhang, S., Benenson, R., Omran, M., Hosang, J., Schiele, B.: How far are we from solving pedestrian detection? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1259–1267 (2016)

  51. Chen, Y., Li, W., Sakaridis, C., Dai, D., VanGool, L.: Domain adaptive faster r-cnn for object detection in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3339–3348 (2018)

  52. Redmon, J., Farhadi, A.: Yolov3: an incremental improvement (2018). arXiv:1804.02767

  53. Zhang, S., Wen, L., Bian, X., Lei, Z., Li, S.Z.: Occlusion-aware r-cnn: detecting pedestrians in a crowd. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 637–653 (2018)

  54. Wang, X., Xiao, T., Jiang, Y., Shao, S., Sun, J., Shen, C.: Repulsion loss: detecting pedestrians in a crowd. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7774–7783 (2018)

  55. Song, T., Sun, L., Xie, D., Sun, H., Pu, S.: Small-scale pedestrian detection based on topological line localization and temporal feature aggregation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 536–551 (2018)

  56. Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., Tian, Q.: Centernet: keypoint triplets for object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 6569–6578 (2019)

  57. Dollár, P., Appel, R., Belongie, R., Perona, P.: Fast feature pyramids for object detection. IEEE Trans. Pattern Anal. Mach. Intell. 36(8), 1532–1545 (2014)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by Key issues of vocational education in Jiangsu Province during the 13th five-year plan of Educational Science (B-b/2020/03/30), and in part by Special subject of Jiangsu Higher Education Society (2020NDKT047).

Author information

Authors and Affiliations

  1. School of Intelligent Manufacturing, Nanjing Vocational College of Information Technology, Nanjing, Jiangsu, China

    Xiaoqin Li

Authors
  1. Xiaoqin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoqin Li.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Human participants

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Animals studies

This article does not contain any studies with animals performed by any of the authors.

Informed consent

This article does not contain any studies with animals performed by any of the authors. Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X. Real-time digital twins end-to-end multi-branch object detection with feature level selection for healthcare. J Real-Time Image Proc 19, 921–930 (2022). https://doi.org/10.1007/s11554-022-01233-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-022-01233-z

Keywords

Search

Navigation