International Journal of Computer Vision

, Volume 117, Issue 2, pp 142–158 | Cite as

Data-Dependent Higher-Order Clique Selection for Artery–Vein Segmentation by Energy Minimization

  • Yoshiro KitamuraEmail author
  • Yuanzhong Li
  • Wataru Ito
  • Hiroshi Ishikawa


We propose a novel segmentation method based on energy minimization of higher-order potentials. We introduce higher-order terms into the energy to incorporate prior knowledge on the shape of the segments. The terms encourage certain sets of pixels to be entirely in one segment or the other. The sets can for instance be smooth curves in order to help delineate pulmonary vessels, which are known to run in almost straight lines. The higher-order terms can be converted to submodular first-order terms by adding auxiliary variables, which can then be globally minimized using graph cuts. We also determine the weight of these terms, or the degree of the aforementioned encouragement, in a principled way by learning from training data with the ground truth. We demonstrate the effectiveness of the method in a real-world application in fully-automatic pulmonary artery–vein segmentation in CT images.


Segmentation Higher-order energy Artery–vein segmentation Surgery simulation 



We appreciate the advice regarding clinical knowledge and subjective tests given by the Department of Radiology, NTT East Sapporo Hospital. This research was done in cooperation with the Department of Thoracic Surgery, Tokyo Medical University Hospital. In the course of this work, Hiroshi Ishikawa was partially supported by the Grant-in-Aid for Scientific Research on Innovative Areas #26108003 from the Japan Society for the Promotion of Science (JSPS) as well as the CREST grant from the Japan Science and Technology Agency (JST).


  1. Boykov, Y., & Kolmogorov, V. (2004). An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26, 1124–1137.CrossRefzbMATHGoogle Scholar
  2. Buelow, T., Wiemker, R., Blaffert, T., Lorenz, C., & Renisch, S. (2005). Automatic Extraction of the Pulmonary Artery Tree from Multi-Slice CT Data. In Proceedings SPIE Medical Imaging p. 5746.Google Scholar
  3. Ebrahimdoost, Y., Qanadli, S. D., Nikravanshalmani, A., Ellis, T. J., Shojaee, Z. F., & Dehmeshki, J. (2011). Automatic segmentation of pulmonary artery (PA) in 3D pulmonary CTA images. In Proceedings of DSP.Google Scholar
  4. El-Zehiry, N. Y., & Grady, L. (2010). Fast global optimization of curvature. In Proceedings of CVPR (pp. 3257–3264).Google Scholar
  5. Ferlay, J., Shin, H. R., Bray, F., Forman, D., Mathers, C., & Parkin, D. M. (2010). Estimates of worldwide burden of cancer in 2008: GLOBOCAN 2008. International Journal of Cancer, 127, 2893–2917.CrossRefGoogle Scholar
  6. Frangi, A. F., Niessen, W. J., Vincken, K. L., & Viergever, M. A. (1998). Multiscale vessel enhancement filtering. In Proceedings of MICCAI (Vol. 1496, pp. 130–137).Google Scholar
  7. Friedman, J., Hastie, T., & Tibshirani, R. (2000). Additive logistic regression: A statistical view of boosting (with discussion and a rejoinder by the authors). Annals of Statistics, 28(2), 337–407.MathSciNetCrossRefzbMATHGoogle Scholar
  8. Gao, Z., Grout, R. W., Holtze, C., Hoffman, E. A., & Saha, P. K. (2012). A new paradigm of interactive artery/vein separation in noncontrast pulmonary CT imaging using multiscale topomorphologic opening. IEEE Transactions on Biomedical Engineering, 59, 3016–3027.CrossRefGoogle Scholar
  9. Hammer, P. L. (1965). Some network flow problems solved with pseudo-boolean programming. Operations Research, 13, 388–399.MathSciNetCrossRefGoogle Scholar
  10. Hammer, P. L., Hansen, P., & Simeone, B. (1984). Roof duality, complementation and persistency in quadratic 0–1 optimization. Mathematical Programming, 28, 121–155.MathSciNetCrossRefzbMATHGoogle Scholar
  11. Ikeda, N., Yoshimura, A., Hagiwara, M., Akata, S., & Saji, H. (2013). Three dimensional computed tomography lung modeling is useful in simulation and navigation of lung cancer surgery. Annals of Thoracic and Cardiovascular Surgery, 19, 1–5.CrossRefGoogle Scholar
  12. Inoue, T., Kitamura, Y., Li, Y., & Ito, W. (2013). Robust airway extraction based on machine learning and minimum spanning tree. In Proceedings of SPIE Medical Imaging (p. 86700L).Google Scholar
  13. Ishikawa, H. (2011). Transformation of general binary MRF minimization to the first order case. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33, 1234–1249.CrossRefGoogle Scholar
  14. Kadoury, S., Abi-Jaoudeh, N., & Valdes, P. A. (2013). Higher-order CRF tumor segmentation with discriminant manifold potentials. In Proceedings of MICCAI (Vol. 8149, pp. 719–726).Google Scholar
  15. Kitamura, Y., Li, Y., & Ito, W. (2012). Automatic coronary extraction by supervised detection and shape matching. In Proceedings of IEEE ISBI, (pp. 234–237).Google Scholar
  16. Kitamura, Y., Li, Y., Ito, W., & Ishikawa, H. (2013). Adaptive higher-order submodular potentials for pulmonary artery-vein segmentation. In the Fifth International Workshop on Pulmonary Image Analysis, MICCAI2013, Nagoya, Japan.
  17. Kohli, P., & Torr, P. H. S. (2005). Efficiently solving dynamic markov random fields using graph cuts. In Proceedings of ICCV.Google Scholar
  18. Kohli, P., Kumar, M. P., & Torr, P. H. S. (2009a). P3 & Beyond: Move making algorithms for solving higher order functions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31, 1645–1656.CrossRefGoogle Scholar
  19. Kohli, P., Ladickỳ, L., & Torr, P. H. S. (2009b). Robust higher order potentials for enforcing label consistency. International Journal of Computer Vision, 82, 302–324.CrossRefGoogle Scholar
  20. Lei, T., Udupa, J. K., Saha, P. K., & Odhner, D. (2001). Artery-vein separation via MRA-An image processing approach. IEEE Transactions on Medical Imaging, 20, 689–703.CrossRefGoogle Scholar
  21. Lei, T. (2010). Statistics of medical imaging. Boca Raton: CRC Press.Google Scholar
  22. Li, S. Z. (2009). Markov random field modeling in image analysis (2nd ed.). London: Springer.zbMATHGoogle Scholar
  23. Lo, P., van Genneken, B., Reinhardt, J. M., Yavarna, T., et al. (2012). Extraction of airways from CT (EXACT09). IEEE Transactions on Medical Imaging, 31(11), 2093–2107.CrossRefGoogle Scholar
  24. Mekada, Y., Nakamura, S., Ide, I., Murase, H., & Otsuji, H. (2006). Pulmonary artery and vein classification using spatial arrangement features from X-ray CT images. In Proceedings of APCCM (pp. 232–235).Google Scholar
  25. Nieuwenhuis, C., Toeppe, E., Gorelick, L., Veksler, O., & Boykov, Y. (2014). Efficient squared curvature. In Proceedings of CVPR.Google Scholar
  26. Nowozin, S., & Lampert, C. H. (2009), Global connectivity potentials for random field models. In Proceedings of CVPR (pp. 818–825).Google Scholar
  27. Olsson, C., Ulén, J., Boykov, Y., & Kolmogorov, V. (2013). Partial enumeration and curvature regularization. In Proceedings of ICCV (pp. 2936–2943).Google Scholar
  28. Oswald, M. R., Jan Stuhmer, J., & Cremers, D. (2014). Generalized connectivity constraints for spatio-temporal 3D reconstruction. In Proceedings of ECCV (pp. 32–46).Google Scholar
  29. Park, S., Lee, S. M., Kim, N., Seo, J. B., & Shin, H. (2013). Automatic reconstruction of the arterial and venous trees on volumetric chest CT. Medical Physics, 40(7), 071906.CrossRefGoogle Scholar
  30. Rother, C., Kolmogorov, V., Lempitsky, V., & Szummer, M. (2007). Optimizing binary MRFs via extended roof duality. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  31. Russell, C., Metaxas, D., Restif, C., & Torr, P. (2007). Using the \(P^n\) Potts model with learning methods to segment live cell images. In Proceedings of ICCV.Google Scholar
  32. Saha, P. K., Gao, Z., Alford, S. K., Sonka, M., & Hoffman, E. A. (2010). Topomorphologic separation of fused isointensity objects via multiscale opening: Separating arteries and veins in 3-D pulmonary CT. IEEE Transactions on Medical Imaging, 29, 840–851.CrossRefGoogle Scholar
  33. Saji, H., Inoue, T., Kato, Y., Shimada, Y., Hagiwara, M., Kudo, Y., et al. (2013). Virtual segmentectomy based on high-quality three-dimensional lung modelling from computed tomography images. Interactive Cardiovascular and Thoracic Surgery, 17(2), 227–232.CrossRefGoogle Scholar
  34. Scharstein, D., & Pal, C. (2007). Learning conditional random fields for stereo. In Proceedings of CVPR.Google Scholar
  35. Schoenemann, T., Kahl, F., Masnou, S., & Cremers, D. (2012). A linear framework for region-based image segmentation and inpainting involving curvature penalization. International Journal of Computer Vision, 99, 53–68.MathSciNetCrossRefzbMATHGoogle Scholar
  36. Sebbe, R., Gosselin, B., Coche, E., & Macq, B. (2003). Pulmonary arteries segmentation and feature extraction through slice marching. In Proceedings of ProRISC workshop on Circuits, Systems and Signal Processing.Google Scholar
  37. Shekhovtsov, A., Kohli, P., & Rother, C. (2012). Curvature prior for MRF-based segmentation and shape inpainting. In Proceedings of DAGM (pp. 41–51).Google Scholar
  38. Strandmark, P., & Kahl, F. (2011), Curvature regularization for curves and surfaces in a global optimization framework. In Proceedings of EMMCVPR (pp. 205–218).Google Scholar
  39. Stühmer, J., Schröder, P., & Cremers, D. (2013). Tree shape priors with connectivity constraints using convex relaxation on general graphs. In Proceedings of ICCV (pp. 2336–2343).Google Scholar
  40. Sun, D., Roth, S., Lewis, J. P., & Black, M. J. (2008). Learning optical flow. In Proceedings of ECCV, (Vol. 5304, pp. 83–97).Google Scholar
  41. van Bemmel, C. M., Spreeuwers, L. J., Viergever, M. A., & Niessen, W. J. (2003). Level-set-based artery-vein separation in blood pool agent CE-MR angiograms. IEEE Transactions on Medical Imaging, 22, 1224–1234.CrossRefGoogle Scholar
  42. Wang, C., Komodakis, N., & Paragios, N. (2013). Markov random field modeling, inference & learning in computer vision & image understanding: A survey. Computer Vision and Image Understanding, 117(11), 1610–1627. Elsevier.Google Scholar
  43. Wang, C., Li, Y., Ito, W., Shimura, K., & Abe, K. (2009). A machine learning approach to extract spinal column centerline from three-dimensional CT data. In Proceedings of SPIE Medical Imaging (p. 72594T).Google Scholar
  44. Yamaguchi, T., Kitasaka, T., Mori, K., Mekada, Y., Hasegawa, J., Toriwaki, J., & Otsuji, H. (2002). A preliminary study for automated recognition of branches of pulmonary artery and vein using anatomical positional relations from a 3-D chest X-ray CT image, CARS2002, 782–787.Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Yoshiro Kitamura
    • 1
    • 2
    • 3
    Email author
  • Yuanzhong Li
    • 1
  • Wataru Ito
    • 1
  • Hiroshi Ishikawa
    • 2
    • 3
  1. 1.Imaging Technology CenterFujifilm CorporationTokyoJapan
  2. 2.Department of Computer Science and EngineeringWaseda UniversityTokyoJapan
  3. 3.JST CRESTSaitamaJapan

Personalised recommendations