In this paper we address the problem of geometric multi-model fitting using a few weakly annotated data points, which has been little studied so far. In weak annotating (WA), most manual annotations are supposed to be correct yet inevitably mixed with incorrect ones. SuchWA data can naturally arise through interaction in various tasks. For example, in the case of homography estimation, one can easily annotate points on the same plane or object with a single label by observing the image. Motivated by this, we propose a novel method to make full use of WA data to boost multi-model fitting performance. Specifically, a graph for model proposal sampling is first constructed using the WA data, given the prior that WA data annotated with the same weak label has a high probability of belonging to the same model. By incorporating this prior knowledge into the calculation of edge probabilities, vertices (i.e., data points) lying on or near the latent model are likely to be associated and further form a subset or cluster for effective proposal generation. Having generated proposals, a-expansion is used for labeling, and our method in return updates the proposals. This procedure works in an iterative way. Extensive experiments validate our method and show that it produces noticeably better results than state-of-the-art techniques in most cases.
Fischler, M. A.; Bolles, R. C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM Vol. 24, No. 6, 381–395, 1981.
Boykov, Y.; Veksler, O.; Zabih, R. Fast approximate energy minimization via graph cuts. In: Proceedings of the 7th IEEE International Conference on Computer Vision, 377–384, 1999.
Delong, A.; Osokin, A.; Isack, H. N.; Boykov, Y. Fast approximate energy minimization with label costs. International Journal of Computer Vision Vol. 96, No. 1, 1–27, 2012.
Isack, H.; Boykov, Y. Energy-based geometric multimodel fitting. International Journal of Computer Vision Vol. 97, No. 2, 123–147, 2012.
Amayo, P.; Pinies, P.; Paz, L. M.; Newman, P. Geometric multi-model fitting with a convex relaxation algorithm. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8138–8146, 2018.
Pham, T. T.; Chin, T. J.; Yu, J.; Suter, D. The random cluster model for robust geometric fitting. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 8, 1658–1671, 2014.
Chum, O.; Matas, J. Matching with PROSAC: Progressive sample consensus. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, 220–226, 2005.
Nistér, D. Preemptive RANSAC for live structure and motion estimation. Machine Vision and Applications Vol. 16, No. 5, 321–329, 2005.
Brachmann, E.; Krull, A.; Nowozin, S.; Shotton, J.; Michel, F.; Gumhold, S.; Rother, C. DSAC: Differentiable RANSAC for camera localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6684–6692, 2017.
Torr, P. H. S. Geometric motion segmentation and model selection. Philosophical Transactions of the Royal Society of London Series A: Mathematical, Physical and Engineering Sciences Vol. 356, No. 1740, 1321–1340, 1998.
Vincent, E.; Laganiére, R. Detecting planar homographies in an image pair. In: Proceedings of the 2nd International Symposium on Image and Signal Processing and Analysis, 182–187, 2001.
Zuliani, M.; Kenney, C. S.; Manjunath, B. S. The multiRANSAC algorithm and its application to detect planar homographies. In: Proceedings of the IEEE International Conference on Image Processing, Vol. 3, III–153, 2005.
Toldo, R.; Fusiello, A. Robust multiple structures estimation with J-linkage. In: Computer Vision–ECCV 2008. Lecture Notes in Computer Science, Vol. 5302. Forsyth, D.; Torr, P.; Zisserman, A. Eds. Springer Berlin Heidelberg, 537–547, 2008.
Magri, L.; Fusiello, A. T-linkage: A continuous relaxation of J-linkage for multi-model fitting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3954–3961, 2014.
Magri, L.; Fusiello, A. Multiple models fitting as a set coverage problem. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3318–3326, 2016.
Yu, J.; Chin, T. J.; Suter, D. A global optimization approach to robust multi-model fitting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2041–2048, 2011.
Jian, Y. D.; Chen, C. S. Two-view motion segmentation by mixtures of dirichlet process with model selection and outlier removal. In: Proceedings of the IEEE 11th International Conference on Computer Vision, 1–8, 2007.
Nieuwenhuis, C.; Töppe, E.; Cremers, D. A survey and comparison of discrete and continuous multilabel optimization approaches for the potts model. International Journal of Computer Vision Vol. 104, No. 3, 223–240, 2013.
Meer, P. Robust techniques for computer vision. In: Emerging Topics in Computer Vision. Medioni, G.; Kang, S. B. Eds. Prentice Hall, 107–190, 2004.
Chum, O.; Matas, J.; Kittler, J. Locally optimized RANSAC. In: Pattern Recognition. Lecture Notes in Computer Science, Vol. 2781. Michaelis, B.; Krell, G. Eds. Springer Berlin Heidelberg, 236–243, 2003.
Tordoff, B. J.; Murray, D. W. Guided-MLESAC: Faster image transform estimation by using matching priors. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 27, No. 10, 1523–1535, 2005.
Chin, T. J.; Yu, J.; Suter, D. Accelerated hypothesis generation for multi-structure robust fitting. In: Computer Vision–ECCV 2010. Lecture Notes in Computer Science, Vol. 6315. Daniilidis K.; Maragos P.; Paragios N. Eds. Springer Berlin Heidelberg, 533–546, 2010.
Figueiredo, M. A. T.; Jain, A. K. Unsupervised learning of finite mixture models. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 24, No. 3, 381–396, 2002.
Swendsen, R. H.; Wang, J. S. Nonuniversal critical dynamics in Monte Carlo simulations. Physical Review Letters Vol. 58, No. 2, 86, 1987.
Hartley, R.; Zisserman, A. Multiple View Geometry in Computer Vision. Cambridge University Press, 2004.
Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics Vol. 23, No. 3, 309–314, 2004.
Lazic, N.; Givoni, I.; Frey, B.; Aarabi, P. FLoSS: Facility location for subspace segmentation. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 825–832, 2009.
Wong, H. S.; Chin, T.J.; Yu, J.; Suter, D. Dynamic and hierarchical multi-structure geometric model fitting. In: Proceedings of the International Conference on Computer Vision, 1044–1051, 2011.
Pham, T. T.; Chin, T.-J.; Yu, J.; Suter, D. Simultaneous sampling and multi-structure fitting with adaptive reversible jump MCMC. In: Proceedings of the Advances in Neural Information Processing Systems 24, 540–548, 2011.
Adbel-Aziz, Y. I. Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry. In: Proceedings of the ASP Symposium on Close-Range Photogrammetry, 1–18, 1971.
Chao Zhang is supported in part by JSPS KAKENHI Grant JP18K17823. Xuequan Lu is supported in part by Deakin CY01-251301-F003-PJ03906-PG00447.
Chao Zhang received his Ph.D. degree from Iwate University (Japan) in 2017. He is now a full-time assistant professor at the Faculty of Engineering, University of Fukui (Japan). His research interests include computer vision and graphics, mainly focusing on feature matching and vision-based optimization problems. He is a member of the IEEE Computer Society, IEEE Signal Processing Society, ACM, and IEICE.
Xuequan Lu is a lecturer (assistant professor) at Deakin University, Australia. He spent more than two years as a research fellow in Singapore. Prior to that, he earned his Ph.D. degree from Zhejiang University (China) in 2016. His research interests lie mainly in visual computing, in areas such as geometry modeling, processing and analysis, animation and simulation, 2D data processing and analysis. More information can be found at http://www.xuequanlu.com.
Katsuya Hotta received his B.E. degree in 2017 and is now pursuing a Ph.D. degree at the University of Fukui, Japan. His current research focuses primarily on computer vision, mainly in subspace clustering and visual tracking.
Xi Yang is currently a project assistant professor in the Graduate School of Information Science and Technology at The University of Tokyo. He received his B.E. degree from the College of Information Engineering at Northwest A&F University in 2012. He received his M.E. and D.E. degrees from the Graduate School of Engineering, Iwate University. His research interests include geometric processing, visualization, and deep learning.
About this article
Cite this article
Zhang, C., Lu, X., Hotta, K. et al. G2MF-WA: Geometric multi-model fitting with weakly annotated data. Comp. Visual Media 6, 135–145 (2020). https://doi.org/10.1007/s41095-020-0166-8
- geometric multi-model fitting
- weak annotation
- multi-homography detection
- two-view motion segmentation