Simultaneous Edge Alignment and Learning

  • Zhiding YuEmail author
  • Weiyang Liu
  • Yang Zou
  • Chen Feng
  • Srikumar Ramalingam
  • B. V. K. Vijaya Kumar
  • Jan Kautz
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11207)


Edge detection is among the most fundamental vision problems for its role in perceptual grouping and its wide applications. Recent advances in representation learning have led to considerable improvements in this area. Many state of the art edge detection models are learned with fully convolutional networks (FCNs). However, FCN-based edge learning tends to be vulnerable to misaligned labels due to the delicate structure of edges. While such problem was considered in evaluation benchmarks, similar issue has not been explicitly addressed in general edge learning. In this paper, we show that label misalignment can cause considerably degraded edge learning quality, and address this issue by proposing a simultaneous edge alignment and learning framework. To this end, we formulate a probabilistic model where edge alignment is treated as latent variable optimization, and is learned end-to-end during network training. Experiments show several applications of this work, including improved edge detection with state of the art performance, and automatic refinement of noisy annotations.

Supplementary material

474178_1_En_24_MOESM1_ESM.pdf (16.3 mb)
Supplementary material 1 (pdf 16732 KB)


  1. 1.
    Arbeláez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. PAMI 33(5), 898–916 (2011)CrossRefGoogle Scholar
  2. 2.
    Arbeláez, P., Pont-Tuset, J., Barron, J., Marques, F., Malik, J.: Multiscale combinatorial grouping. In: CVPR (2014)Google Scholar
  3. 3.
    Bertasius, G., Shi, J., Torresani, L.: Deepedge: a multiscale bifurcated deep network for top-down contour detection. In: CVPR (2015)Google Scholar
  4. 4.
    Bertasius, G., Shi, J., Torresani, L.: High-for-low, low-for-high: efficient boundary detection from deep object features and its applications to high-level vision. In: ICCV (2015)Google Scholar
  5. 5.
    Bertasius, G., Shi, J., Torresani, L.: Semantic segmentation with boundary neural fields. In: CVPR (2016)Google Scholar
  6. 6.
    Bilmes, J.: On virtual evidence and soft evidence in Bayesian networks. Technical report (2004)Google Scholar
  7. 7.
    Canny, J.: A computational approach to edge detection. IEEE Trans. PAMI 6, 679–698 (1986)CrossRefGoogle Scholar
  8. 8.
    Castrejón, L., Kundu, K., Urtasun, R., Fidler, S.: Annotating object instances with a polygon-RNN. In: CVPR (2017)Google Scholar
  9. 9.
    Chen, L.C., Barron, J.T., Papandreou, G., Murphy, K., Yuille, A.L.: Semantic image segmentation with task-specific edge detection using CNNS and a discriminatively trained domain transform. In: CVPR (2016)Google Scholar
  10. 10.
    Cherkassky, B.V., Goldberg, A.V.: On implementing push-relabel method for the maximum flow problem. In: Balas, E., Clausen, J. (eds.) IPCO 1995. LNCS, vol. 920, pp. 157–171. Springer, Heidelberg (1995). Scholar
  11. 11.
    Cordts, M., et al.: The Cityscapes dataset for semantic urban scene understanding. In: CVPR (2016)Google Scholar
  12. 12.
    Dollar, P., Tu, Z., Belongie, S.: Supervised learning of edges and object boundaries. In: CVPR (2006)Google Scholar
  13. 13.
    Dollár, P., Zitnick, C.L.: Fast edge detection using structured forests. IEEE Trans. PAMI 37(8), 1558–1570 (2015)CrossRefGoogle Scholar
  14. 14.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL visual object classes challenge 2011 (VOC2011) results.
  15. 15.
    Goldberg, A.V., Kennedy, R.: An efficient cost scaling algorithm for the assignment problem. SIAM J. Discrete Math. (1993)Google Scholar
  16. 16.
    Hancock, E.R., Kittler, J.: Edge-labeling using dictionary-based relaxation. IEEE Trans. PAMI 12(2), 165–181 (1990)CrossRefGoogle Scholar
  17. 17.
    Hariharan, B., Arbeláez, P., Bourdev, L., Maji, S., Malik, J.: Semantic contours from inverse detectors. In: ICCV (2011)Google Scholar
  18. 18.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)Google Scholar
  19. 19.
    Hoiem, D., Efros, A.A., Hebert, M.: Geometric context from a single image. In: ICCV (2005)Google Scholar
  20. 20.
    Hwang, J., Liu, T.L.: Pixel-wise deep learning for contour detection. In: ICLR (2015)Google Scholar
  21. 21.
    Karsch, K., Liao, Z., Rock, J., Barron, J.T., Hoiem, D.: Boundary cues for 3D object shape recovery. In: CVPR (2013)Google Scholar
  22. 22.
    Khoreva, A., Benenson, R., Omran, M., Hein, M., Schiele, B.: Weakly supervised object boundaries. In: CVPR (2016)Google Scholar
  23. 23.
    Kittler, J.: On the accuracy of the sobel edge detector. Image Vis. Comput. 1(1), 37–42 (1983)CrossRefGoogle Scholar
  24. 24.
    Kokkinos, I.: Pushing the boundaries of boundary detection using deep learning (2016)Google Scholar
  25. 25.
    Konishi, S., Yuille, A.L., Coughlan, J.M., Zhu, S.C.: Statistical edge detection: learning and evaluating edge cues. IEEE Trans. PAMI 25(1), 57–74 (2003)CrossRefGoogle Scholar
  26. 26.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: NIPS (2012)Google Scholar
  27. 27.
    Li, Y., Yang, J., Song, Y., Cao, L., Luo, J., Li, L.J.: Learning from noisy labels with distillation. In: CVPR (2017)Google Scholar
  28. 28.
    Liao, L., Choudhury, T., Fox, D., Kautz, H.A.: Training conditional random fields using virtual evidence boosting. In: IJCAI (2007)Google Scholar
  29. 29.
    Lim, J., Zitnick, C., Dollar, P.: Sketch tokens: a learned mid-level representation for contour and object detection. In: CVPR (2013)Google Scholar
  30. 30.
    Liu, Y., Cheng, M.M., Hu, X., Wang, K., Bai, X.: Richer convolutional features for edge detection. In: CVPR (2017)Google Scholar
  31. 31.
    Maire, M., Yu, S.X., Perona, P.: Reconstructive sparse code transfer for contour detection and semantic labeling. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 273–287. Springer, Cham (2015). Scholar
  32. 32.
    Malik, J.: Interpreting line drawings of curved objects. Int. J. Comput. Vis. 1(1), 73–103 (1987)CrossRefGoogle Scholar
  33. 33.
    Malik, J., Maydan, D.: Recovering three-dimensional shape from a single image of curved objects. IEEE Trans. PAMI 11(6), 555–566 (1989)CrossRefGoogle Scholar
  34. 34.
    Martin, D.R., Fowlkes, C.C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Trans. PAMI 26(5), 530–549 (2004)CrossRefGoogle Scholar
  35. 35.
    Patrini, G., Rozza, A., Menon, A.K., Nock, R., Qu, L.: Making deep neural networks robust to label noise: a loss correction approach. In: CVPR (2017)Google Scholar
  36. 36.
    Pearl, J.: Probabilistic reasoning in intelligent systems: networks of plausible inference (1988)CrossRefGoogle Scholar
  37. 37.
    Pinheiro, P.O., Collobert, R., Dollár, P.: Learning to segment object candidates. In: NIPS (2015)Google Scholar
  38. 38.
    Prasad, M., Zisserman, A., Fitzgibbon, A., Kumar, M.P., Torr, P.H.S.: Learning class-specific edges for object detection and segmentation. In: Kalra, P.K., Peleg, S. (eds.) ICVGIP 2006. LNCS, vol. 4338, pp. 94–105. Springer, Heidelberg (2006). Scholar
  39. 39.
    Ren, X., Fowlkes, C.C., Malik, J.: Learning probabilistic models for contour completion in natural images. Int. J. Comput. Vis. 77(1–3), 47–63 (2008)CrossRefGoogle Scholar
  40. 40.
    Rupprecht, C., Huaroc, E., Baust, M., Navab, N.: Deep active contours. arXiv preprint arXiv:1607.05074 (2016)
  41. 41.
    Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: Labelme: a database and web-based tool for image annotation. IJCV 77(1–3), 157–173 (2008)CrossRefGoogle Scholar
  42. 42.
    Shan, Q., Curless, B., Furukawa, Y., Hernandez, C., Seitz, S.: Occluding contours for multi-view stereo. In: CVPR (2014)Google Scholar
  43. 43.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)Google Scholar
  44. 44.
    Sugihara, K.: Machine Interpretation of Line Drawings. MIT Press, Wiley (1986)Google Scholar
  45. 45.
    Vahdat, A.: Toward robustness against label noise in training deep discriminative neural networks. In: NIPS (2017)Google Scholar
  46. 46.
    Veit, A., Alldrin, N., Chechik, G., Krasin, I., Gupta, A., Belongie, S.: Learning from noisy large-scale datasets with minimal supervision. In: CVPR (2017)Google Scholar
  47. 47.
    Wang, Y., et al.: Iterative learning with open-set noisy labels. In: CVPR (2018)Google Scholar
  48. 48.
    Xiao, T., Xia, T., Yang, Y., Huang, C., Wang, X.: Learning from massive noisy labeled data for image classification. In: CVPR (2015)Google Scholar
  49. 49.
    Xie, S., Tu, Z.: Holistically-nested edge detection. In: ICCV (2015)Google Scholar
  50. 50.
    Yang, J., Price, B., Cohen, S., Lee, H., Yang, M.H.: Object contour detection with a fully convolutional encoder-decoder network. In: CVPR (2016)Google Scholar
  51. 51.
    Yu, Z., Liu, W., Liu, W., Peng, X., Hui, Z., Kumar, B.V.: Generalized transitive distance with minimum spanning random forest. In: IJCAI (2015)Google Scholar
  52. 52.
    Yu, Z., Feng, C., Liu, M.Y., Ramalingam, S.: CaseNet: deep category-aware semantic edge detection. In: CVPR (2017)Google Scholar
  53. 53.
    Zitnick, C.L., Dollár, P.: Edge boxes: locating object proposals from edges. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 391–405. Springer, Cham (2014). Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Zhiding Yu
    • 1
    Email author
  • Weiyang Liu
    • 3
  • Yang Zou
    • 2
  • Chen Feng
    • 4
  • Srikumar Ramalingam
    • 5
  • B. V. K. Vijaya Kumar
    • 2
  • Jan Kautz
    • 1
  1. 1.NVIDIASanta ClaraUSA
  2. 2.Carnegie Mellon UniversityPittsburghUSA
  3. 3.Georgia Institute of TechnologyAtlantaGeorgia
  4. 4.New York UniversityNew York CityUSA
  5. 5.University of UtahSalt Lake CityUSA

Personalised recommendations