International Journal of Computer Vision

, Volume 45, Issue 2, pp 129–156 | Cite as

Three-Dimensional Reconstruction of Points and Lines with Unknown Correspondence across Images

  • Y.-Q. Cheng
  • X.G. Wang
  • R.T. Collins
  • E.M. Riseman
  • A.R. Hanson


Three-dimensional reconstruction from a set of images is an important and difficult problem in computer vision. In this paper, we address the problem of determining image feature correspondences while simultaneously reconstructing the corresponding 3D features, given the camera poses of disparate monocular views. First, two new affinity measures are presented that capture the degree to which candidate features from different images consistently represent the projection of the same 3D point or 3D line. An affinity measure for point features in two different views is defined with respect to their distance from a hypothetical projected 3D pseudo-intersection point. Similarly, an affinity measure for 2D image line segments across three views is defined with respect to a 3D pseudo-intersection line. These affinity measures provide a foundation for determining unknown correspondences using weighted bipartite graphs representing candidate point and line matches across different images. As a result of this graph representation, a standard graph-theoretic algorithm can provide an optimal, simultaneous matching and triangulation of points across two views, and lines across three views. Experimental results on synthetic and real data demonstrate the effectiveness of the approach.

feature correspondence matching point/line affinity measure weighted bipartite graph matching maximum network flow 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Amerinex Artificial Intelligence. 1991. The KBVision System User's Guide. AAI: Amherst, MA.Google Scholar
  2. Aggarwal, J.K., Davis, L.S., and Martin, W.N. 1981. Correspondence processes in dynamic scene analysis. In Proc. of IEEE 69, pp. 562–572.Google Scholar
  3. Aloimonos, J. and Rigoutsos, I. 1986. Determining 3-D motion of a rigid surface patch without correspondences under perspective projection. In Proc. of AAAI, pp. 681–688.Google Scholar
  4. ARPA. 1994. RADIUS, Section III. In Arpa Image Understanding Workshop. Monterey, CA.Google Scholar
  5. Ayache, N. 1991. Artificial Vision for Mobile Robots: Stereo Vision and Multisensory Perception. The MIT Press: Cambridge, MA.Google Scholar
  6. Basu, A. and Aloimonos J. 1987. A robust algorithm for determining the translation of a rigidly moving surface without correspondence for robotics applications. In Proc. of IJCAI, pp. 815–818.Google Scholar
  7. Bedekar, A.S. and Haralick, R.M. 1996. Finding corresponding points based on Bayesian triangulation. In Proc. of CVPR'96, San Francisco, CA, pp. 61–66.Google Scholar
  8. Blostein, S.D. and Huang, T.S. 1987. Quantization errors in stereo triangulation. In Proc. of First International Conf. on Computer Vision, pp. 325–334.Google Scholar
  9. Boldt, M., Weiss, R., and Riseman, E.M. 1989. Token-based extraction of straight lines. IEEE Trans. SMC, 19(6):1581–1594.Google Scholar
  10. Burns, J.B., Hanson, A.R., and Riseman, E.M. 1986. Extracting straight lines. IEEE Trans. PAMI, 8(4):425–455.Google Scholar
  11. Chang, S.F. and McCormick, S.T. 1989. A fast implementation of a bipartite matching algorithm. Technical Report, Columbia University, New York.Google Scholar
  12. Cheng, Y.Q. 1996. Acquisition of 3D Models from a Set of 2D Images. Ph.D. Thesis, CMPSCI TR96–91, Computer Science Department, University of Massachusetts.Google Scholar
  13. Cheng, Y.Q., Collins, R., Hanson, A.R., and Riseman, E.M. 1994. Triangulation without correspondences. Arpa Image Understanding Workshop, Monterey, CA, pp. 993–1000.Google Scholar
  14. Cheng, Y.Q., Wu, V., Collins, R.T., Hanson, A.R., and Riseman, E.M. 1996. Maximum-weight bipartite matching technique and its application in image feature matching. In Proc. SPIE VCIP.Google Scholar
  15. Cheriyan, J. and Maheshwari, S.N. 1989. Analysis of preflowpush algorithms for maximum network flow. SIAM J. Comput., 18:1057–1086.Google Scholar
  16. Chou, T.C. and Kanatani, K. 1987. Recovering 3D rigid motions without correspondences. In Proc. ICCV, pp. 534–538.Google Scholar
  17. Collins, R.T. 1996. A space-sweep approach to true multi-image matching. In Proc. of CVPR'96, San Francisco, CA, pp. 358–363.Google Scholar
  18. Collins, R.T., Cheng, Y.Q., Jayes, C., Stolle, F., Wang, X.G., Hanson, A.R., and Riseman, E.M. 1998. The ASCENDER system automated site modeling from multiple aerial images. Computer Vision and Image Understanding, 72(2):143–162.Google Scholar
  19. Collins, R.T., Cheng, Y.Q., Jayes, C., Stolle, F., Wang, X.G., Hanson, A.R., and Riseman, E.M. 1996. Site model acquisition and Three-Dimensional Reconstruction of Points and Lines 155 extension from aerial images. In Proc. ICCV, IEEE, pp. 888–893.Google Scholar
  20. Collins, R.T., Hanson, A.R., Riseman, E.M., and Cheng, Y.Q. 1993. Model matching and extension for automated 3D modeling. In Proc. IUW, pp. 197–203.Google Scholar
  21. Collins, R.T., Hanson, A.R., Riseman, E.M., Jayes, C.O., Stolle, F., Wang, X.G., and Cheng, Y.Q. 1996. UMass progress in 3D building model acquisition. In Arpa Image Understanding Workshop, Palm Springs, CA, pp. 305–316.Google Scholar
  22. Collins, R.T., Jayes, C.O., Stolle, F., Wang, X.G., Cheng, Y.Q., Hanson, A.R., and Riseman, E.M. 1995. A system for automated site model acquisition. In Integrating Photogrammetric Techniques with Scene Analysis and Machine Vision II, SPIE Proceeding. Vol. 7617, Orlando, FL, pp. 244–254.Google Scholar
  23. Deriche, R., Vaillant, R., and Faugeras, O.D. 1992. From noisy edge points to 3D reconstruction of a scene: A robust approach and its uncertainty analysis. In Proc. of the Second Europeian Conf. on Computer Vision.Google Scholar
  24. Edmonds, J. 1965. Maximum matching and polyhedron with 0,1 vertices. J. Research of the National Bureau of Standards, 69B.Google Scholar
  25. Edmonds, J. and Karp, R.M. 1972. Theoretical improvements in algorithmic efficiency for network flowproblems. J. Assoc. Comput. Math., 19:248–264.Google Scholar
  26. Ford, L.R. and Fulkerson, E. 1962. Flows in Networks. Princeton Univ. Press: Princeton, NJ.Google Scholar
  27. Gabow, H.N. 1985. A scaling algorithm for weighted matching on general graphs. In Proc. 26th Annual Symp. of the Foundations of Computer Science, pp. 90–100.Google Scholar
  28. Gabow, H.N. 1990. Data structure for weighted matching and nearest common ancestors with linking. Proceedings 1st Annual ACMSIAM Symposium on Discrete Algorithms, pp. 434–443.Google Scholar
  29. Ganapathy, S. 1984. Decomposition of transformation matrices for robot vision. In Proc. IEEE Int. Conf. Robotics and Automation, Atlanta, GA, pp. 130–139.Google Scholar
  30. Gold, S. and Rangarajan, A. 1996. Graph matching by graduated assignment. In Proc. of CVPR'96, San Francisco, CA, pp. 239–244.Google Scholar
  31. Goldberg, A.V. 1987. Efficient graph algorithms for sequential and parallel computers. Ph.D thesis, MIT, Cambridge, MA.Google Scholar
  32. Goldberg, A.V. and Tarjan, R.E. 1988. A new approach to the maximum-flow problem. J. Assoc. Comput. Mach., 35:921–940.Google Scholar
  33. Goldgof, D.B., Lee, H., and Huang, T.S. 1992. Matching and motion estimation of three-dimensional point and line sets using eigenstructure without correspondences. Pattern Recognition, 25(3):271–286.Google Scholar
  34. Griffin, P.M. 1989. Correspondence of 2-D projections by bipartite matching. Pattern Recognition Letter, 9:361–366.Google Scholar
  35. Gruen, A. and Baltsavias, E. 1988. Geometrically constrained multiphoto matching. Photogrammetric Engineering and Remote Sensing, 54(6):633–641.Google Scholar
  36. Hao, J. and Kocur, G. 1993. An implementation of a shortest augmenting path algorithm for the assignment problem. Network flows and matching: First DIMACS Implementation Challenge, Series in DIMACS, American Mathematical Society: Rhode Island, Vol. 12, pp. 453–468.Google Scholar
  37. Hartley, R.I. 1994. Triangulation. In Proc. IUW, pp. 957–966.Google Scholar
  38. Hopcroft, J.E. and Karp, R.M. 1973. An n 2.5 algorithm for maximum matching in bipartite graphs. J. ACM, 23:225–231.Google Scholar
  39. Ito, E. and Aloimonos, J. 1988. Is correspondence necessary for the perception of structure from motion?. In Image Understanding Workshop, pp. 921–929.Google Scholar
  40. Jaynes, C., Stoll, F., and Collins, R. 1994. Task driven perceptual organization for extraction of rooftop polygons. In Proceeding ARPA Image Understanding Workshop, Monterey, CA, pp. 359–365.Google Scholar
  41. Johnson, D.S. and McGeoch, C.C. 1993. Network flows and matching: First DIMACS Implementation Challenge, Series in DIMACS, American Mathematical Society: R.I., Vol. 12.Google Scholar
  42. Jonker, R. and Volgenant, A. 1987. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing, 38:325–340.Google Scholar
  43. Kim, W. and Kak, A. 1991. 3D object recognition using bipartite matching embedded in discrete relaxation. IEEE PAMI, 13(3):224–251.Google Scholar
  44. Kahn, P., Kitchen, L., and Riseman, E. 1990. Real-time feature extraction: A fast line finder for vision-guided robot navigation. IEEE Pattern Analysis and machine Intelligence, 12(11):1098–1102.Google Scholar
  45. Kuhn, H.W. 1955. The Hungarian method for assignment problem. Naval Research Logistics, Quarterly, Vol. 2, pp. 83–97.Google Scholar
  46. Kumar, R. 1992. Model dependent Inference of 3D information from a sequence of 2D images. Ph.D. Thesis, CMPSCI TR92–04, Computer Science Department, University of Massachusetts.Google Scholar
  47. Kumar, R. and Hanson, A.R. 1992a. Model extension and refinement using pose recovery techniques. Journal of Robotic Systems, 9(6):753–771.Google Scholar
  48. Kumar, R. and Hanson, A.R. 1992b. Application of pose determination techniques to model extension and refinement. Proceedings Darpa Image UnderstandingWorkshop, San Diego, CA, pp. 727–744.Google Scholar
  49. Kumar, R., Anandan, P., and Hanna, K. 1994. Shape recovery from multiple views: A parallax based approach. In Proc. of ARPA Image Understanding, Monterey, CA, pp. 947–955.Google Scholar
  50. Lee, C.-H. and Joshi, A. 1993. Correspondence problem in image sequence analysis. Pattern Recognition, 26(1):47–61.Google Scholar
  51. Lee, H., Lin, Z., and Huang, T.S. 1986. Finding 3-D point correspondences in motion estimation. In Proc. of Eighth Int. Conf. on Pattern Recognition, pp. 303–305.Google Scholar
  52. Lee, H., Lin, Z., and Huang, T.S. 1990. Estimating rigid-body motion from three-dimentional data without matching point correspondences. Int. J. Imaging Systems Technol., 2:55–62.Google Scholar
  53. Lessard, R., Rousseau, J.-M., and Minoux, M. 1989.Anewalgorithm for general matching problems using network flow subproblems. Networks, 19:459–479.Google Scholar
  54. Martin Marietta and SRI International. 1993. RCDE User's Guide. Martin Marietta, Management and Data Systems: Philadelpha, PA.Google Scholar
  55. Micali, S. and Vazirani, V. 1980. An \(O\left( {\sqrt {\left| V \right|.\left| E \right|} } \right)\)algorithm for finding maximum matchings in general graphs. In Proc. 21st Symp. Foundations of Computer Science, pp. 17–27.Google Scholar
  56. Pentland, A. and Horowitz, B. 1991. Recovery of non-rigid motion and structure. IEEE PAMI, 13(7):730–742.Google Scholar
  57. Pentland, A. and Sclaroff, S. 1991. Closed-form solutions for physically-based shape modeling and recognition. IEEE PAMI, 13(7):715–729.Google Scholar
  58. Roy, S. and Cox, I.J. 1998. A maximum-flow formulation of the Ncamera stereo correspondence problem. In ICCV98, pp. 492–499.Google Scholar
  59. Sclaroff, S. and Pentland, A. 1993. A modal framework for correspondence and description. In Proc. of IEEE, pp. 308–313.Google Scholar
  60. Scott, G.L. and Longuet-Higgins, H.C. 1991. An algorithm for associating the features of two patterns. Proc. Roy Soc Lond, B244:21–26.Google Scholar
  61. Shapiro, L.S. and Brady, J.M. 1992. Feature-based correspondence: An eigenvector approach. Image and Vision Computing, 10(5):283–288.Google Scholar
  62. Ullman, S. 1979. The Interpretation of Visual Motion.MIT Press: Cambridge, MA.Google Scholar
  63. Wang, X., Cheng, Y.Q., Collins, R.T., and Hanson, A.R. 1996. Determining correspondences and rigid motion of 3-D point sets with missing data. In Proc. CVPR, IEEE, pp. 252–257.Google Scholar
  64. Wu, Z. and Leahy, R. 1993. An optimal graph theoretic approach to data clustering: Theory and its application to image segmentation. IEEE PAMI, 13(7):1101–1113.Google Scholar
  65. Wu, M.S. and Leou, J.J. 1995. A bipartite matching approach to feature correspondence in stereo vision. Pattern Recognition Letter, 16:23–31.Google Scholar
  66. Zhang, Z. and Faugeras, O.D. 1990a. Building a 3D world model with a mobile robot: 3D line segment representation and integration. In Proc. of the IEEE International Conf. on Pattern Recognition, Atlantic City, New Jersey, pp. 38–42.Google Scholar
  67. Zhang, Z. and Faugeras, O.D. 1990b. Tracking and grouping 3D line segments. In Proc. of 3rd International Conf. on Computer Vision, Osaka, Japan, pp. 577–580.Google Scholar

Copyright information

© Kluwer Academic Publishers 2001

Authors and Affiliations

  • Y.-Q. Cheng
    • 1
  • X.G. Wang
    • 1
  • R.T. Collins
    • 2
  • E.M. Riseman
    • 2
  • A.R. Hanson
    • 2
  1. 1.Robotics Institute, NSHCarnegie Mellon UniversityPittsburghUSA
  2. 2.Department of Computer ScienceUniversity of MassachusettsAmherstUSA

Personalised recommendations