Count- and Similarity-Aware R-CNN for Pedestrian Detection

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12362)


Recent pedestrian detection methods generally rely on additional supervision, such as visible bounding-box annotations, to handle heavy occlusions. We propose an approach that leverages pedestrian count and proposal similarity information within a two-stage pedestrian detection framework. Both pedestrian count and proposal similarity are derived from standard full-body annotations commonly used to train pedestrian detectors. We introduce a count-weighted detection loss function that assigns higher weights to the detection errors occurring at highly overlapping pedestrians. The proposed loss function is utilized at both stages of the two-stage detector. We further introduce a count-and-similarity branch within the two-stage detection framework, which predicts pedestrian count as well as proposal similarity. Lastly, we introduce a count and similarity-aware NMS strategy to identify distinct proposals. Our approach requires neither part information nor visible bounding-box annotations. Experiments are performed on the CityPersons and CrowdHuman datasets. Our method sets a new state-of-the-art on both datasets. Further, it achieves an absolute gain of 2.4% over the current state-of-the-art, in terms of log-average miss rate, on the heavily occluded (HO) set of CityPersons test set. Finally, we demonstrate the applicability of our approach for the problem of human instance segmentation. Code and models are available at:


Pedestrian detection Human instance segmentation 



The work is supported by the National Key R&D Program of China (Grant # 2018AAA0102800 and 2018AAA0102802) and National Natural Science Foundation of China (Grant # 61632018).

Supplementary material

504472_1_En_6_MOESM1_ESM.pdf (14.9 mb)
Supplementary material 1 (pdf 15283 KB)


  1. 1.
    Bodla, N., Singh, B., Chellappa, R., Davis, L.S.: Soft-NMS - improving object detection with one line of code. In: ICCV (2017)Google Scholar
  2. 2.
    Brazil, G., Yin, X., Liu, X.: Illuminating pedestrians via simultaneous detection & segmentation. In: ICCV (2017)Google Scholar
  3. 3.
    Cai, Z., Fan, Q., Feris, R.S., Vasconcelos, N.: A unified multi-scale deep convolutional neural network for fast object detection. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 354–370. Springer, Cham (2016). Scholar
  4. 4.
    Cai, Z., Vasconcelos, N.: Cascade r-cnn: High quality object detection and instance segmentation. arXiv preprint arXiv:1906.09756 (2019)
  5. 5.
    Cao, J., Pang, Y., Han, J., Gao, B., Li, X.: Taking a look at small-scale pedestrians and occluded pedestrians. IEEE Trans. Image Process. 29, 3143–3152 (2020)CrossRefGoogle Scholar
  6. 6.
    Cao, J., Pang, Y., Zhao, S., Li, X.: High-level semantic networks for multi-scale object detection. IEEE Trans. Circ. Syst. Video Technol. 30, 3372–3386 (2019)CrossRefGoogle Scholar
  7. 7.
    Chi, C., Zhang, S., Xing, J., Lei, Z., Li, S.Z.X.Z.: Pedhunter: occlusion robust pedestrian detector in crowded scenes. In: AAAI (2020)Google Scholar
  8. 8.
    Dollar, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: an evaluation of the state of the art. TPAMI 34, 743–761 (2012)CrossRefGoogle Scholar
  9. 9.
    He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: ICCV (2017)Google Scholar
  10. 10.
    Hosang, J., Benenson, R., Schiele, B.: Learning non-maximum suppression. In: CVPR (2017)Google Scholar
  11. 11.
    Hu, H., Gu, J., Zhang, Z., Dai, J., Wei, Y.: Relation networks for object detection. In: CVPR (2018)Google Scholar
  12. 12.
    Jiang, B., Luo, R., Mao, J., Xiao, T., Jiang, Y.: Acquisition of localization confidence for accurate object detection. In: ECCV (2018)Google Scholar
  13. 13.
    Liu, S., Huang, D., Wang, Y.: Adaptive NMS: refining pedestrian detection in a crowd. In: CVPR (2019)Google Scholar
  14. 14.
    Liu, W., Liao, S., Hu, W., Liang, X., Chen, X.: Learning efficient single-stage pedestrian detectors by asymptotic localization fitting. In: ECCV (2018)Google Scholar
  15. 15.
    Liu, W., Liao, S., Ren, W., Hu, W., Yu, Y.: High-level semantic feature detection: a new perspective for pedestrian detection. In: CVPR (2019)Google Scholar
  16. 16.
    Mao, J., Xiao, T., Jiang, Y., Cao, Z.: What can help pedestrian detection? In: CVPR (2017)Google Scholar
  17. 17.
    Mathias, M., Benenson, R., Timofte, R., Gool, L.V.: Handling occlusions with franken-classifiers. In: ICCV (2013)Google Scholar
  18. 18.
    Nie, J., Anwer, R.M., Cholakkal, H., Khan, F.S., Pang, Y., Shao, L.: Enriched feature guided refinement network for object detection. In: ICCV (2019)Google Scholar
  19. 19.
    Ouyang, W., Wang, X.: Joint deep learning for pedestrian detection. In: ICCV (2013)Google Scholar
  20. 20.
    Pang, Y., Xie, J., Khan, M.H., Anwer, R.M., Khan, F.S., Shao, L.: Mask-Guided attention network for occluded pedestrian detection. In: ICCV (2019)Google Scholar
  21. 21.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NIPS (2015)Google Scholar
  22. 22.
    Shao, S., et al.: Crowdhuman: A benchmark for detecting human in a crowd. arXiv preprint arXiv:1805.00123 (2018)
  23. 23.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  24. 24.
    Song, T., Sun, L., Xie, D., Sun, H., Pu, S.: Small-scale pedestrian detection based on topological line localization and temporal feature aggregation. In: ECCV (2018)Google Scholar
  25. 25.
    Tian, Y., Luo, P., Wang, X., Tang, X.: Deep learning strong parts for pedestrian detection. In: ICCV (2015)Google Scholar
  26. 26.
    Tychsen-Smith, L., Petersson, L.: Improving object localization with fitness nms and bounded IOU loss. In: CVPR (2018)Google Scholar
  27. 27.
    Wang, X., Xiao, T., Jiang, Y., Shao, S., Sun, J., Shen, C.: Repulsion loss: detecting pedestrians in a crowd. In: CVPR (2018)Google Scholar
  28. 28.
    Xie, J., Pang, Y., Cholakkal, H., Anwer, R.M., Khan, F.S., Shao, L.: PSC-net: learning part spatial co-occurrence for occluded pedestrian detection. arXiv preprint arXiv:2001.09252 (2020)
  29. 29.
    Zhang, J., et al.: Attribute-aware pedestrian detection in a crowd. arXiv preprint arXiv:1910.09188 (2019)
  30. 30.
    Zhang, S., Benenson, R., Schiele, B.: Citypersons: a diverse dataset for pedestrian detection. In: CVPR (2017)Google Scholar
  31. 31.
    Zhang, S., Yang, J., Schiele, B.: Occluded pedestrian detection through guided attention in CNNs. In: CVPR (2018)Google Scholar
  32. 32.
    Zhang, S., Wen, L., Bian, X., Lei, Z., Li, S.Z.: Occlusion-aware R-CNN: detecting pedestrians in a crowd. In: ECCV (2018)Google Scholar
  33. 33.
    Zhang, S.H., et al.: Pose2seg: detection free human instance segmentation. In: CVPR (2019)Google Scholar
  34. 34.
    Zhou, C., Yang, M., Yuan, J.: Discriminative feature transformation for occluded pedestrian detection. In: ICCV (2019)Google Scholar
  35. 35.
    Zhou, C., Yuan, J.: Non-rectangular part discovery for object detection. In: BMVC (2014)Google Scholar
  36. 36.
    Zhou, C., Yuan, J.: Multi-label learning of part detectors for heavily occluded pedestrian detection. In: ICCV (2017)Google Scholar
  37. 37.
    Zhou, C., Yuan, J.: Bi-box regression for pedestrian detection and occlusion estimation. In: ECCV (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Tianjin Key Laboratory of Brain-Inspired Artificial Intelligence, School of Electrical and Information EngineeringTianjin UniversityTianjinChina
  2. 2.Mohamed bin Zayed University of Artificial IntelligenceAbu DhabiUAE
  3. 3.Inception Institute of Artificial IntelligenceAbu DhabiUAE
  4. 4.University of Central FloridaOrlandoUSA

Personalised recommendations