Skip to main content
Download PDF

G2MF-WA: Geometric multi-model fitting with weakly annotated data

Computational Visual Media volume 6, pages 135–145 (2020)Cite this article

Abstract

In this paper we address the problem of geometric multi-model fitting using a few weakly annotated data points, which has been little studied so far. In weak annotating (WA), most manual annotations are supposed to be correct yet inevitably mixed with incorrect ones. SuchWA data can naturally arise through interaction in various tasks. For example, in the case of homography estimation, one can easily annotate points on the same plane or object with a single label by observing the image. Motivated by this, we propose a novel method to make full use of WA data to boost multi-model fitting performance. Specifically, a graph for model proposal sampling is first constructed using the WA data, given the prior that WA data annotated with the same weak label has a high probability of belonging to the same model. By incorporating this prior knowledge into the calculation of edge probabilities, vertices (i.e., data points) lying on or near the latent model are likely to be associated and further form a subset or cluster for effective proposal generation. Having generated proposals, a-expansion is used for labeling, and our method in return updates the proposals. This procedure works in an iterative way. Extensive experiments validate our method and show that it produces noticeably better results than state-of-the-art techniques in most cases.

References

  1. Fischler, M. A.; Bolles, R. C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM Vol. 24, No. 6, 381–395, 1981.

    Article  MathSciNet  Google Scholar 

  2. Boykov, Y.; Veksler, O.; Zabih, R. Fast approximate energy minimization via graph cuts. In: Proceedings of the 7th IEEE International Conference on Computer Vision, 377–384, 1999.

    Google Scholar 

  3. Delong, A.; Osokin, A.; Isack, H. N.; Boykov, Y. Fast approximate energy minimization with label costs. International Journal of Computer Vision Vol. 96, No. 1, 1–27, 2012.

    Article  MathSciNet  MATH  Google Scholar 

  4. Isack, H.; Boykov, Y. Energy-based geometric multimodel fitting. International Journal of Computer Vision Vol. 97, No. 2, 123–147, 2012.

    Article  MATH  Google Scholar 

  5. Amayo, P.; Pinies, P.; Paz, L. M.; Newman, P. Geometric multi-model fitting with a convex relaxation algorithm. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8138–8146, 2018.

    Google Scholar 

  6. Pham, T. T.; Chin, T. J.; Yu, J.; Suter, D. The random cluster model for robust geometric fitting. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 8, 1658–1671, 2014.

    Article  Google Scholar 

  7. Chum, O.; Matas, J. Matching with PROSAC: Progressive sample consensus. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, 220–226, 2005.

    Google Scholar 

  8. Nistér, D. Preemptive RANSAC for live structure and motion estimation. Machine Vision and Applications Vol. 16, No. 5, 321–329, 2005.

    Article  Google Scholar 

  9. Brachmann, E.; Krull, A.; Nowozin, S.; Shotton, J.; Michel, F.; Gumhold, S.; Rother, C. DSAC: Differentiable RANSAC for camera localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6684–6692, 2017.

    Google Scholar 

  10. Torr, P. H. S. Geometric motion segmentation and model selection. Philosophical Transactions of the Royal Society of London Series A: Mathematical, Physical and Engineering Sciences Vol. 356, No. 1740, 1321–1340, 1998.

    Article  MathSciNet  MATH  Google Scholar 

  11. Vincent, E.; Laganiére, R. Detecting planar homographies in an image pair. In: Proceedings of the 2nd International Symposium on Image and Signal Processing and Analysis, 182–187, 2001.

    Google Scholar 

  12. Zuliani, M.; Kenney, C. S.; Manjunath, B. S. The multiRANSAC algorithm and its application to detect planar homographies. In: Proceedings of the IEEE International Conference on Image Processing, Vol. 3, III–153, 2005.

    Google Scholar 

  13. Toldo, R.; Fusiello, A. Robust multiple structures estimation with J-linkage. In: Computer Vision–ECCV 2008. Lecture Notes in Computer Science, Vol. 5302. Forsyth, D.; Torr, P.; Zisserman, A. Eds. Springer Berlin Heidelberg, 537–547, 2008.

    Google Scholar 

  14. Magri, L.; Fusiello, A. T-linkage: A continuous relaxation of J-linkage for multi-model fitting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3954–3961, 2014.

    Google Scholar 

  15. Magri, L.; Fusiello, A. Multiple models fitting as a set coverage problem. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3318–3326, 2016.

    Google Scholar 

  16. Yu, J.; Chin, T. J.; Suter, D. A global optimization approach to robust multi-model fitting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2041–2048, 2011.

    Google Scholar 

  17. Jian, Y. D.; Chen, C. S. Two-view motion segmentation by mixtures of dirichlet process with model selection and outlier removal. In: Proceedings of the IEEE 11th International Conference on Computer Vision, 1–8, 2007.

    Google Scholar 

  18. Nieuwenhuis, C.; Töppe, E.; Cremers, D. A survey and comparison of discrete and continuous multilabel optimization approaches for the potts model. International Journal of Computer Vision Vol. 104, No. 3, 223–240, 2013.

    Article  MathSciNet  MATH  Google Scholar 

  19. Meer, P. Robust techniques for computer vision. In: Emerging Topics in Computer Vision. Medioni, G.; Kang, S. B. Eds. Prentice Hall, 107–190, 2004.

    Google Scholar 

  20. Chum, O.; Matas, J.; Kittler, J. Locally optimized RANSAC. In: Pattern Recognition. Lecture Notes in Computer Science, Vol. 2781. Michaelis, B.; Krell, G. Eds. Springer Berlin Heidelberg, 236–243, 2003.

    Google Scholar 

  21. Tordoff, B. J.; Murray, D. W. Guided-MLESAC: Faster image transform estimation by using matching priors. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 27, No. 10, 1523–1535, 2005.

    Article  Google Scholar 

  22. Chin, T. J.; Yu, J.; Suter, D. Accelerated hypothesis generation for multi-structure robust fitting. In: Computer Vision–ECCV 2010. Lecture Notes in Computer Science, Vol. 6315. Daniilidis K.; Maragos P.; Paragios N. Eds. Springer Berlin Heidelberg, 533–546, 2010.

    Google Scholar 

  23. Figueiredo, M. A. T.; Jain, A. K. Unsupervised learning of finite mixture models. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 24, No. 3, 381–396, 2002.

    Article  Google Scholar 

  24. Swendsen, R. H.; Wang, J. S. Nonuniversal critical dynamics in Monte Carlo simulations. Physical Review Letters Vol. 58, No. 2, 86, 1987.

    Article  Google Scholar 

  25. Hartley, R.; Zisserman, A. Multiple View Geometry in Computer Vision. Cambridge University Press, 2004.

    Book  MATH  Google Scholar 

  26. Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics Vol. 23, No. 3, 309–314, 2004.

    Article  Google Scholar 

  27. Lazic, N.; Givoni, I.; Frey, B.; Aarabi, P. FLoSS: Facility location for subspace segmentation. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 825–832, 2009.

    Google Scholar 

  28. Wong, H. S.; Chin, T.J.; Yu, J.; Suter, D. Dynamic and hierarchical multi-structure geometric model fitting. In: Proceedings of the International Conference on Computer Vision, 1044–1051, 2011.

    Google Scholar 

  29. Pham, T. T.; Chin, T.-J.; Yu, J.; Suter, D. Simultaneous sampling and multi-structure fitting with adaptive reversible jump MCMC. In: Proceedings of the Advances in Neural Information Processing Systems 24, 540–548, 2011.

    Google Scholar 

  30. Adbel-Aziz, Y. I. Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry. In: Proceedings of the ASP Symposium on Close-Range Photogrammetry, 1–18, 1971.

    Google Scholar 

Download references

Acknowledgements

Chao Zhang is supported in part by JSPS KAKENHI Grant JP18K17823. Xuequan Lu is supported in part by Deakin CY01-251301-F003-PJ03906-PG00447.

Author information

Affiliations

  1. University of Fukui, Fukui, 910-8507, Japan

    Chao Zhang & Katsuya Hotta

  2. Deakin University, Waurn Ponds, 3216, Australia

    Xuequan Lu

  3. The University of Tokyo, Tokyo, 113-8656, Japan

    Xi Yang

Authors
  1. Chao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuequan Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Katsuya Hotta
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xi Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chao Zhang.

Additional information

Chao Zhang received his Ph.D. degree from Iwate University (Japan) in 2017. He is now a full-time assistant professor at the Faculty of Engineering, University of Fukui (Japan). His research interests include computer vision and graphics, mainly focusing on feature matching and vision-based optimization problems. He is a member of the IEEE Computer Society, IEEE Signal Processing Society, ACM, and IEICE.

Xuequan Lu is a lecturer (assistant professor) at Deakin University, Australia. He spent more than two years as a research fellow in Singapore. Prior to that, he earned his Ph.D. degree from Zhejiang University (China) in 2016. His research interests lie mainly in visual computing, in areas such as geometry modeling, processing and analysis, animation and simulation, 2D data processing and analysis. More information can be found at http://www.xuequanlu.com.

Katsuya Hotta received his B.E. degree in 2017 and is now pursuing a Ph.D. degree at the University of Fukui, Japan. His current research focuses primarily on computer vision, mainly in subspace clustering and visual tracking.

Xi Yang is currently a project assistant professor in the Graduate School of Information Science and Technology at The University of Tokyo. He received his B.E. degree from the College of Information Engineering at Northwest A&F University in 2012. He received his M.E. and D.E. degrees from the Graduate School of Engineering, Iwate University. His research interests include geometric processing, visualization, and deep learning.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhang, C., Lu, X., Hotta, K. et al. G2MF-WA: Geometric multi-model fitting with weakly annotated data. Comp. Visual Media 6, 135–145 (2020). https://doi.org/10.1007/s41095-020-0166-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41095-020-0166-8

Keywords

  • geometric multi-model fitting
  • weak annotation
  • multi-homography detection
  • two-view motion segmentation