3D Line Segment Reconstruction in Structured Scenes via Coplanar Line Segment Clustering

  • Kai Li
  • Jian YaoEmail author
  • Li Li
  • Yahui Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10117)


This paper presents a new algorithm aiming for 3D Line Segment (LS) reconstruction in structured scenes that are comprised of a set of planes. Due to location imprecision of image LSs, it often produces many erroneous reconstructions when reconstructing 3D LSs by triangulating corresponding LSs from two images. We propose to solve this problem by first recovering space planes and then back-projecting image LSs onto the recovered space planes to get reliable 3D LSs. Given LS matches identified from two images, we estimate a set of planar homographies and use them to cluster the LS matches into groups such that LS matches in each group are related by the same homography induced by a space plane. In each LS match group, the corresponding space plane can be recovered from the 3D LSs obtained by triangulating all the LS correspondences. To reduce the incidence of incorrect LS match grouping, we formulate to solve the LS match grouping problem into solving a multi-label optimization problem. The advantages of the proposed algorithm over others in this area are that it can generate more complete and detailed 3D models of scenes using much fewer images and can recover the space planes where the reconstructed 3D LSs lie, which is beneficial for upper level applications, like scene understanding and building facade extraction.


Line Segment Point Match Space Plane Reconstruction Accuracy Adjacency Graph 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was partially supported by the National Natural Science Foundation of China (Project No. 41571436), the National Natural Science Foundation of China under Grant 91438203, the Hubei Province Science and Technology Support Program, China (Project No. 2015BAA027), the Jiangsu Province Science and Technology Support Program, China (Project No. BE2014866), and the South Wisdom Valley Innovative Research Team Program.


  1. 1.
    Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S.M., Szeliski, R.: Building Rome in a day. Commun. ACM 54, 105–112 (2011)CrossRefGoogle Scholar
  2. 2.
    Baillard, C., Schmid, C., Zisserman, A., Fitzgibbon, A.: Automatic line matching and 3D reconstruction of buildings from multiple views. In: ISPRS Conference on Automatic Extraction of GIS Objects from Digital Imagery (1999)Google Scholar
  3. 3.
    Bartoli, A., Sturm, P.: Structure-from-motion using lines: representation, triangulation, and bundle adjustment. Comput. Vis. Image Underst. 100, 416–441 (2005)CrossRefGoogle Scholar
  4. 4.
    Bay, H., Ess, A., Neubeck, A., Van Gool, L.: 3D from line segments in two poorly-textured, uncalibrated images. In: 3DPVT (2006)Google Scholar
  5. 5.
    Boykov, Y., Veksler, O., Zabih, R.: Efficient approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 36, 1222–1239 (2001)CrossRefGoogle Scholar
  6. 6.
    Chetverikov, D., Svirko, D., Stepanov, D., Krsek, P.: The trimmed iterative closest point algorithm. In: ICPR (2002)Google Scholar
  7. 7.
    Delmerico, J.A., David, P., Corso, J.J.: Building facade detection, segmentation, and parameter estimation for mobile robot stereo vision. Image Vis. Comput. 31, 841–852 (2013)CrossRefGoogle Scholar
  8. 8.
    Delong, A., Osokin, A., Isack, H.N., Boykov, Y.: Fast approximate energy minimization with label costs. Int. J. Comput. Vis. 96, 1–27 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. 32, 1362–1376 (2010)CrossRefGoogle Scholar
  10. 10.
    Habib, A.F., Morgan, M., Lee, Y.R.: Bundle adjustment with selfcalibration using straight lines. Photogram. Rec. 17, 635–650 (2002)CrossRefGoogle Scholar
  11. 11.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)zbMATHGoogle Scholar
  12. 12.
    Hofer, M., Maurer, M., Bischof, H.: Improving sparse 3D models for man-made environments using line-based 3D reconstruction. In: 3DV (2014)Google Scholar
  13. 13.
    Hofer, M., Wendel, A., Bischof, H.: Incremental line-based 3D reconstruction using geometric constraints. In: BMVC (2013)Google Scholar
  14. 14.
    Hofer, M., Maurer, M., Bischof, H.: Efficient 3D scene abstraction using line segments. Comput. Vis. Image Underst. (2016). doi: 10.1016/j.cviu.2016.03.017
  15. 15.
    Li, K., Yao, J., Lu, X., Xia, M., Li, L.: Joint point and line segment matching on wide-baseline stereo images. In: WACV (2016)Google Scholar
  16. 16.
    Kim, C., Manduchi, R.: Planar structures from line correspondences in a Manhattan World. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9003, pp. 509–524. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-16865-4_33 Google Scholar
  17. 17.
    Jain, A., Kurz, C., Thormahlen, T., Seidel, H.P.: Exploiting global connectivity constraints for reconstruction of 3D line segments from images. In: CVPR (2010)Google Scholar
  18. 18.
    Jensen, R., Dahl, A., Vogiatzis, G., Tola, E.: Large scale multi-view stereopsis evaluation. In: CVPR (2014)Google Scholar
  19. 19.
    Luong, Q.-T., Viéville, T.: Canonical representations for the geometries of multiple projective views. Comput. Vis. Image Underst. 64, 193–229 (1996)CrossRefGoogle Scholar
  20. 20.
    Matinec, D., Pajdla, T.: Line reconstruction from many perspective images by factorization. In: CVPR (2003)Google Scholar
  21. 21.
    Micusik, B., Wildenauer, H.: Structure from motion with line segments under relaxed endpoint constraints. In: 3DV (2014)Google Scholar
  22. 22.
    Micusik, B., Wildenauer, H.: Descriptor free visual indoor localization with line segments. In: 3DV (2015)Google Scholar
  23. 23.
    Pan, J.: Coherent scene understanding with 3D geometric reasoning. Ph.D. thesis, Carnegie Mellon University (2014)Google Scholar
  24. 24.
    Pham, T.T., Chin, T.J., Yu, J., Suter, D.: The random cluster model for robust geometric fitting. IEEE Trans. Pattern Anal. Mach. Intell. 36, 1658–1671 (2014)CrossRefGoogle Scholar
  25. 25.
    Přibyl, B., Zemčík, P., Čadík, M.: Camera pose estimation from lines using Plücker coordinates. In: BMVC (2015)Google Scholar
  26. 26.
    Ramalingam, S., Brand, M.: Lifting 3D Manhattan lines from a single image. In: ICCV (2013)Google Scholar
  27. 27.
    Schindler, G., Krishnamurthy, P., Dellaert, F.: Line-based structure from motion for urban environments. In: 3DPVT (2006)Google Scholar
  28. 28.
    Sinha, S.N., Steedly, D., Szeliski, R.: Piecewise planar stereo for image-based rendering. In: ICCV (2009)Google Scholar
  29. 29.
    Smith, P., Reid, I.D., Davison, A.J.: Real-time monocular SLAM with straight lines. In: BMVC (2006)Google Scholar
  30. 30.
    Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. ACM Trans. Graph. 25, 835–846 (2006)CrossRefGoogle Scholar
  31. 31.
    Snavely, N., Seitz, S.M., Szeliski, R.: Modeling the world from internet photo collections. Int. J. Comput. Vis. 80, 189–210 (2008)CrossRefGoogle Scholar
  32. 32.
    Spetsakis, M.E., Aloimonos, J.Y.: Structure from motion using line correspondences. Int. J. Comput. Vis. 4, 171–183 (1990)CrossRefGoogle Scholar
  33. 33.
    Strecha, C., Hansen, W.V., Gool, L.V., Fua, P., Thoennessen, U.: On benchmarking camera calibration and multi-view stereo for high resolution imagery. In: CVPR (2008)Google Scholar
  34. 34.
    Taylor, C.J., Kriegman, D.J.: Structure and motion from line segments in multiple images. IEEE Trans. Pattern Anal. Mach. Intell. 17, 1021–1032 (1995)CrossRefGoogle Scholar
  35. 35.
    Teboul, O., Simon, L., Koutsourakis, P., Paragios, N.: Segmentation of building facades using procedural shape priors. In: CVPR (2010)Google Scholar
  36. 36.
    Werner, T., Zisserman, A.: New techniques for automated architectural reconstruction from photographs. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 541–555. Springer, Heidelberg (2002). doi: 10.1007/3-540-47967-8_36 CrossRefGoogle Scholar
  37. 37.
    Wu, C.: Towards linear-time incremental structure from motion. In: 3DV (2013)Google Scholar
  38. 38.
    Zhang, L., Koch, R.: Structure and motion from line correspondences: representation, projection, initialization and sparse bundle adjustment. J. Vis. Commun. Image Represent. 25, 904–915 (2014)CrossRefGoogle Scholar
  39. 39.
    Zhang, Z.: Flexible camera calibration by viewing a plane from unknown orientations. In: ICCV (1999)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.School of Remote Sensing and Information EngineeringWuhan UniversityWuhanPeople’s Republic of China

Personalised recommendations