Rethinking Class Activation Mapping for Weakly Supervised Object Localization

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12360)


Weakly supervised object localization (WSOL) is a task of localizing an object in an image only using image-level labels. To tackle the WSOL problem, most previous studies have followed the conventional class activation mapping (CAM) pipeline: (i) training CNNs for a classification objective, (ii) generating a class activation map via global average pooling (GAP) on feature maps, and (iii) extracting bounding boxes by thresholding based on the maximum value of the class activation map. In this work, we reveal the current CAM approach suffers from three fundamental issues: (i) the bias of GAP that assigns a higher weight to a channel with a small activation area, (ii) negatively weighted activations inside the object regions and (iii) instability from the use of the maximum value of a class activation map as a thresholding reference. They collectively cause the problem that the localization to be highly limited to small regions of an object. We propose three simple but robust techniques that alleviate the problems, including thresholded average pooling, negative weight clamping, and percentile as a standard for thresholding. Our solutions are universally applicable to any WSOL methods using CAM and improve their performance drastically. As a result, we achieve the new state-of-the-art performance on three benchmark datasets of CUB-200–2011, ImageNet-1K, and OpenImages30K.


Weakly Supervised Object Localization (WSOL) Class Activation Mapping (CAM) 



We appreciate Hyunwoo Kim and Jinhwan Seo for their valuable comments. This work was supported by AIR Lab (AI Research Lab) in Hyundai Motor Company through HMC-SNU AI Consortium Fund, and the ICT R&D program of MSIT/IITP (No. 2019-0-01309, Development of AI technology for guidance of a mobile robot to its goal with uncertain maps in indoor/outdoor environments and No.2019-0-01082, SW StarLab).

Supplementary material

504470_1_En_37_MOESM1_ESM.pdf (10.6 mb)
Supplementary material 1 (pdf 10897 KB)


  1. 1.
    Bazzani, L., Bergamo, A., Anguelov, D., Torresani, L.: Self-Taught Object Localization With Deep Networks. In: WACV (2016)Google Scholar
  2. 2.
    Benenson, R., Popov, S., Ferrari, V.: Large-scale Interactive Object Segmentation with Human Annotators. In: CVPR (2019)Google Scholar
  3. 3.
    Chattopadhay, A., Sarkar, A., Howlader, P., Balasubramanian, V.N.: Grad-Cam++: Improved Visual Explanations for Deep Convolutional Networks. In: WACV (2018)Google Scholar
  4. 4.
    Choe, J., Oh, S.J., Lee, S., Chun, S., Akata, Z., Shim, H.: Evaluating Weakly Supervised Object Localization Methods Right. In: CVPR (2020)Google Scholar
  5. 5.
    Choe, J., Park, J.H., Shim, H.: Improved Techniques for Weakly-Supervised Object Localization. Arxiv:1802.07888 (2018)
  6. 6.
    Choe, J., Shim, H.: Attention-Based Dropout Layer for Weakly Supervised Object Localization. In: CVPR (2019)Google Scholar
  7. 7.
    Dai, J., Li, Y., He, K., Sun, J.: R-FCN: Object Detection via Region-Based Fully Convolutional Networks. In: NeurIPS (2016)Google Scholar
  8. 8.
    Durand, T., Mordan, T., Thome, N., Cord, M.: WILDCAT: Weakly Supervised Learning of Deep ConvNets for Image Classification, Pointwise Localization and Segmentation. In: CVPR (2017)Google Scholar
  9. 9.
    Durand, T., Thome, N., Cord, M.: WELDON: Weakly Supervised Learning of Deep Convolutional Neural Networks. In: CVPR (2016)Google Scholar
  10. 10.
    Everingham, M., Eslami, S.M.A., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The Pascal Visual Object Classes Challenge: A Retrospective. IJCV (2015)Google Scholar
  11. 11.
    He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: ICCV (2017)Google Scholar
  12. 12.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: CVPR (2016)Google Scholar
  13. 13.
    Hou, Q., Jiang, P., Wei, Y., Cheng, M.M.: Self-Erasing Network for Integral Object Attention. In: NeurIPS (2018)Google Scholar
  14. 14.
    Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. Arxiv:1704.04861 (2017)
  15. 15.
    Hu, J., Shen, L., Sun, G.: Squeeze-and-Excitation Networks. In: CVPR (2018)Google Scholar
  16. 16.
    Ioffe, S., Szegedy, C.: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In: ICML (2015)Google Scholar
  17. 17.
    Kim, D., Cho, D., Yoo, D., Kweon, I.: Two-Phase Learning for Weakly Supervised Object Localization. In: ICCV (2017)Google Scholar
  18. 18.
    Li, K., Wu, Z., Peng, K.C., Ernst, J., Fu, Y.: Tell Me Where to Look: Guided Attention Inference Network. In: CVPR (2018)Google Scholar
  19. 19.
    Lin, M., Chen, Q., Yan, S.: Network in Network. In: ICLR (2014)Google Scholar
  20. 20.
    Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal Loss for Dense Object Detection. In: ICCV (2017)Google Scholar
  21. 21.
    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common Objects in Context. In: ECCV (2014)Google Scholar
  22. 22.
    Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: SSD: Single Shot Multibox Detector. In: ECCV (2016)Google Scholar
  23. 23.
    Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Is Object Localization for Free? - Weakly-Supervised Learning With Convolutional Neural Networks. In: CVPR (2015)Google Scholar
  24. 24.
    Pinheiro, P.O., Collobert, R.: From Image-Level to Pixel-Level Labeling With Convolutional Networks. In: CVPR (2015)Google Scholar
  25. 25.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You Only Look Once: Unified, Real-Time Object Detection. In: CVPR (2016)Google Scholar
  26. 26.
    Redmon, J., Farhadi, A.: YOLO9000: Better, Faster. CVPR, Stronger. In (2017)Google Scholar
  27. 27.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In: NeurIPS (2015)Google Scholar
  28. 28.
    Russakovsky, O., Deng, J., SU, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet Large Scale Visual Recognition Challenge. IJCV (2015)Google Scholar
  29. 29.
    Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-Cam: Visual Explanations From Deep Networks via Gradient-Based Localization. In: ICCV (2017)Google Scholar
  30. 30.
    Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. Arxiv:1409.1556 (2014)
  31. 31.
    Singh, K.K., Lee, Y.J.: Hide-And-Seek: Forcing a Network to Be Meticulous for Weakly-Supervised Object and Action Localization. In: ICCV (2017)Google Scholar
  32. 32.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going Deeper with Convolutions. In: CVPR (2015)Google Scholar
  33. 33.
    Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The Caltech-UCSD Birds-200-2011 Dataset. Tech. Rep. Cns-Tr-2011-001, California Institute of Technology (2011)Google Scholar
  34. 34.
    Wei, Y., Feng, J., Liang, X., Cheng, M.M., Zhao, Y., Yan, S.: Object Region Mining With Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. In: CVPR (2017)Google Scholar
  35. 35.
    Wei, Y., Shen, Z., Cheng, B., Shi, H., Xiong, J., Feng, J., Huang, T.: TS2C: Tight Box Mining with Surrounding Segmentation Context for Weakly Supervised Object Detection. In: ECCV (2018)Google Scholar
  36. 36.
    Xue, H., Wan, F., Jiao, J., Ji, X., Qixiang, Y.: DANet: Divergent Activation for Weakly supervised Object Localization. In: ICCV (2019)Google Scholar
  37. 37.
    Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In: ICCV (2019)Google Scholar
  38. 38.
    Zhang, X., Wei, Y., Feng, J., Yang, Y., Huang, T.S.: Adversarial Complementary Learning for Weakly Supervised Object Localization. In: CVPR (2018)Google Scholar
  39. 39.
    Zhang, X., Wei, Y., Kang, G., Yang, Y., Huang, T.: Self-Produced Guidance for Weakly-Supervised Object Localization. In: ECCV (2018)Google Scholar
  40. 40.
    Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning Deep Features for Discriminative Localization. In: CVPR (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringSeoul National UniversitySeoulKorea

Personalised recommendations