Semantic Image Segmentation Using Visible and Near-Infrared Channels

  • Neda Salamati
  • Diane Larlus
  • Gabriela Csurka
  • Sabine Süsstrunk
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7584)


Recent progress in computational photography has shown that we can acquire physical information beyond visible (RGB) image representations. In particular, we can acquire near-infrared (NIR) cues with only slight modification to any standard digital camera. In this paper, we study whether this extra channel can improve semantic image segmentation. Based on a state-of-the-art segmentation framework and a novel manually segmented image database that contains 4-channel images (RGB+NIR), we study how to best incorporate the specific characteristics of the NIR response. We show that it leads to improved performances for 7 classes out of 10 in the proposed dataset and discuss the results with respect to the physical properties of the NIR response.


Conditional Random Field Regularization Part Fisher Vector Pairwise Potential Conditional Random Field Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Shotton, J., Winn, J.M., Rother, C., Criminisi, A.: TextonBoost: Joint Appearance, Shape and Context Modeling for Multi-class Object Recognition and Segmentation. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006, Part I. LNCS, vol. 3951, pp. 1–15. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  2. 2.
    Csurka, G., Perronnin, F.: An efficient approach to semantic segmentation. IJCV 95 (2011)Google Scholar
  3. 3.
    Verbeek, J., Triggs, B.: Region classification with markov field aspects models. In: CVPR (2007)Google Scholar
  4. 4.
    Ladicky, L., Russell, C., Kohli, P., Torr, P.H.S.: Graph Cut Based Inference with Co-occurrence Statistics. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 239–253. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  5. 5.
    Finlayson, G.D., Drew, M.S., Funt, B.V.: Color constancy: generalized diagonal transforms suffice. Journal of the Optical Society of America 11, 3011–3019 (1994)CrossRefGoogle Scholar
  6. 6.
    Fredembach, C., Süsstrunk, S.: Colouring the Near-infrared. In: CIC (2008)Google Scholar
  7. 7.
    Schaul, L., Fredembach, C., Süsstrunk, S.: Color image dehazing using the Near-infrared. In: ICIP (2009)Google Scholar
  8. 8.
    Krishnan, D., Fergus, F.: Dark flash photography. In: SIGGRAPH (2009)Google Scholar
  9. 9.
    Brown, M., Süsstrunk, S.: Multispectral SIFT for scene category recognition. In: CVPR (2011)Google Scholar
  10. 10.
    Salamati, N., Larlus, D., Csurka, G.: Combining visible and Near-infrared cues for image categorisation. In: BMVC (2011)Google Scholar
  11. 11.
    Salamati, N., Fredembach, C., Süsstrunk, S.: Material classification using color and NIR images. In: CIC (2009)Google Scholar
  12. 12.
    Zhou, W., Huang, G., Troy, A., Cadenasso, M.L.: Object-based land cover classification of shaded areas in high spatial resolution imagery of urban areas: A comparison study. Remote Sensing of Environment 113, 1769–1777 (2009)CrossRefGoogle Scholar
  13. 13.
    Zhang, X., Sim, T., Miao, X.: Enhancing photographs with NIR images. In: CVPR (2008)Google Scholar
  14. 14.
    Kermani, Z., Lu, Y., Süsstrunk, S.: Correlation-based joint acquisition and demosaicing of visible and Near-infrared images. In: ICIP (2011)Google Scholar
  15. 15.
    Lafferty, J., McCallum, A., Pereira, F.: Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: ICML (2001)Google Scholar
  16. 16.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. IJCV (2004)Google Scholar
  17. 17.
    Csurka, G., Dance, C., Fan, L., Willamowski, J., Bray, C.: Visual categorization with bags of keypoints. In: ECCV SLCV Workshop (2004)Google Scholar
  18. 18.
    Perronnin, F., Dance, C.: Fisher kernels on visual vocabularies for image categorization. In: CVPR (2007)Google Scholar
  19. 19.
    Salamati, N., Süsstrunk, S.: Material-based object segmentation using Near-infrared Information. In: CIC (2010)Google Scholar
  20. 20.
    Rother, C., Kolmogorov, V., Blake, A.: “GrabCut”: interactive foreground extraction using iterated graph cuts. In: SIGGRAPH (2004)Google Scholar
  21. 21.
    Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE TPAMI 26, 1124–1137 (2004)CrossRefGoogle Scholar
  22. 22.
    Kolmogorov, V., Zabih, R.: What energy functions can be minimized via Graph Cuts? IEEE TPAMI 26, 147–159 (2004)CrossRefGoogle Scholar
  23. 23.
    van de Sande, K.E.A., Gevers, T., Snoek, C.G.M.: Evaluating color descriptors for object and scene recognition. IEEE TPAMI 32, 1582–1596 (2010)CrossRefGoogle Scholar
  24. 24.
    Kohli, P., Ladický, L., Torr, P.H.S.: Robust higher order potentials for enforcing label consistency. IJCV 82, 302–324 (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Neda Salamati
    • 1
    • 2
  • Diane Larlus
    • 2
  • Gabriela Csurka
    • 2
  • Sabine Süsstrunk
    • 1
  1. 1.IVRG, IC, École Polytechnique Fédérale de LausanneSwitzerland
  2. 2.Xerox Research Centre EuropeMeylanFrance

Personalised recommendations