Skip to main content
SpringerLink
Log in

Point sets joint registration and co-segmentation

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

We present a novel approach of joint registration and co-segmentation for point sets where objects move in different ways. We consider joint registration and co-segmentation as two problems that are heavily entangled with each other; thus, we represent the input point sets as samples from a generative model and bring up with a novel formulation based on Gaussian mixture model. By maximizing the posterior probability of the samples, we gradually recover the latent object models as well as an object-level segmentation and simultaneously align the segmented points to the latent object models. Along with the formulation, we design an interactive tool that helps users intuitively intervene the process to optimize the registration and segmentation results. The experiment results on a group of synthetic and scanned point clouds demonstrate that our method is powerful and effective for joint registration and co-segmentation on point sets of multiple objects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Armeni, I., Sener, O., Zamir, A.R., Jiang, H., Brilakis, I., Fischer, M., Savarese, S.: 3D semantic parsing of large-scale indoor spaces. In: IEEE Computer Society, pp. 1534–1543. Los Alamitos, CA (2016). https://doi.org/10.1109/CVPR.2016.170

  2. Campbell, D., Petersson, L.: Gogma: Globally-optimal Gaussian mixture alignment. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5685–5694 (2016). https://doi.org/10.1109/CVPR.2016.613

  3. Charles, R.Q., Su, H., Kaichun, M., Guibas, L.J.: Pointnet: deep learning on point sets for 3d classification and segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77–85 (2017). https://doi.org/10.1109/CVPR.2017.16

  4. Chen, H., Bhanu, B.: 3D free-form object recognition in range images using local surface patches. Pattern Recogn. Lett. 28(10), 1252–1262 (2007). https://doi.org/10.1016/j.patrec.2007.02.009

    Article  Google Scholar 

  5. Chen, K., Lai, Y.K., Wu, Y.X., Martin, R., Hu, S.M.: Automatic semantic modeling of indoor scenes from low-quality RGB-D data using contextual information. ACM Trans. Graph. 33(6), 208:1–208:12 (2014). https://doi.org/10.1145/2661229.2661239

    Article  Google Scholar 

  6. Corsini, M., Cignoni, P., Scopigno, R.: Efficient and flexible sampling with blue noise properties of triangular meshes. IEEE Trans. Vis. Comput. Graph. 18(6), 914–924 (2012). https://doi.org/10.1109/TVCG.2012.34

    Article  Google Scholar 

  7. Dai, A., Nießner, M., Zollhöfer, M., Izadi, S., Theobalt, C.: Bundlefusion: real-time globally consistent 3D reconstruction using online surface re-integration. ACM Trans. Graph. 36(4), 76 (2017). https://doi.org/10.1145/3072959.3126814

    Article  Google Scholar 

  8. Dema, M.A., Sari-Sarraf, H.: 3D scene generation by learning from examples. In: IEEE International Symposium on Multimedia, pp. 58–64 (2012). https://doi.org/10.1109/ISM.2012.19

  9. Evangelidis, G.D., Kounades-Bastian, D., Horaud, R., Psarakis, E.Z.: A generative model for the joint registration of multiple point sets. In: ECCV, pp. 109–122 (2014). https://doi.org/10.1007/978-3-319-10584-0_8

    Chapter  Google Scholar 

  10. Fisher, M., Ritchie, D., Savva, M., Funkhouser, T., Hanrahan, P.: Example-based synthesis of 3d object arrangements. ACM Trans. Graph. 31(6), 135:1–135:11 (2012). https://doi.org/10.1145/2366145.2366154

    Article  Google Scholar 

  11. Fisher, M., Savva, M., Li, Y., Hanrahan, P., Nießner, M.: Activity-centric scene synthesis for functional 3D scene modeling. ACM Trans. Graph. 34(6), 179:1–179:13 (2015). https://doi.org/10.1145/2816795.2818057

    Article  Google Scholar 

  12. Guo, H., Zhu, D., Mordohai, P.: Correspondence estimation for non-rigid point clouds with automatic part discovery. Vis. Comput. 32(12), 1511–1524 (2016). https://doi.org/10.1007/s00371-015-1136-5

    Article  Google Scholar 

  13. Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., Fitzgibbon, A.: Kinectfusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST ’11, pp. 559–568. ACM, New York (2011). https://doi.org/10.1145/2047196.2047270

  14. Jia, Z., Gallagher, A.C., Saxena, A., Chen, T.: 3D reasoning from blocks to stability. IEEE Trans. Pattern Anal. Mach. Intell. 37(5), 905–918 (2015). https://doi.org/10.1109/TPAMI.2014.2359435

    Article  Google Scholar 

  15. Jian, B., Vemuri, B.C.: Robust point set registration using Gaussian mixture models. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1633–1645 (2011). https://doi.org/10.1109/TPAMI.2010.223

    Article  Google Scholar 

  16. Li, Y., Paluri, M., Rehg, J.M., Dollar, P.: Unsupervised learning of edges. In: CVPR, pp. 1619–1627 (2016). https://doi.org/10.1109/CVPR.2016.179

  17. Liu, Z., Zhang, Y., Wu, W., Liu, K., Sun, Z.: Model-driven indoor scenes modeling from a single image. In: Proceedings of the 41st Graphics Interface Conference, GI ’15, Halifax, Nova Scotia, Canada, June 3–5, 2015, pp. 25–32. Canadian Information Processing Society, Toronto (2015). http://dl.acm.org/citation.cfm?id=2788896

  18. Merrell, P., Schkufza, E., Li, Z., Agrawala, M., Koltun, V.: Interactive furniture layout using interior design guidelines. ACM Trans. Graph. 30(4), 87:1–87:10 (2011). https://doi.org/10.1145/2010324.1964982

    Article  Google Scholar 

  19. Myronenko, A., Song, X.: Point set registration: coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2262–2275 (2010). https://doi.org/10.1109/TPAMI.2010.46

    Article  Google Scholar 

  20. Nan, L., Xie, K., Sharf, A.: A search-classify approach for cluttered indoor scene understanding. ACM Trans. Graph. 31(6), 137:1–137:10 (2012). https://doi.org/10.1145/2366145.2366156

    Article  Google Scholar 

  21. Nießner, M., Zollhöfer, M., Izadi, S., Stamminger, M.: Real-time 3d reconstruction at scale using voxel hashing. ACM Trans. Graph. 32(6), 169:1–169:11 (2013). https://doi.org/10.1145/2508363.2508374

    Article  Google Scholar 

  22. Rother, C., Kolmogorov, V., Blake, A.: GrabCut: interactive foreground extraction using iterated graph cuts. In: ACM SIGGRAPH 2004 Papers, pp. 309–314. ACM, New York (2004). https://doi.org/10.1145/1186562.1015720

  23. Rother, C., Minka, T.P., Blake, A., Kolmogorov, V.: Cosegmentation of image pairs by histogram matching: incorporating a global constraint into MRFS. In: CVPR, pp. 993–1000 (2006). https://doi.org/10.1109/CVPR.2006.91

  24. Strasdat, H., Newcombe, R.A., Salas-Moreno, R.F., Kelly, P.H., Davison, A.J.: Slam++: simultaneous localisation and mapping at the level of objects. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pp. 1352–1359 (2013). https://doi.org/10.1109/CVPR.2013.178

  25. Taniai, T., Sinha, S.N., Sato, Y.: Joint recovery of dense correspondence and cosegmentation in two images. In: CVPR, pp. 4246–4255 (2016). https://doi.org/10.1109/CVPR.2016.460

  26. Tombari, F., Stefano, L.D.: Object recognition in 3d scenes with occlusions and clutter by hough voting. In: Fourth Pacific-Rim Symposium on Image and Video Technology, pp. 349–355 (2010). https://doi.org/10.1109/PSIVT.2010.65

  27. Wan, L., Zou, C., Zhang, H.: Full and partial shape similarity through sparse descriptor reconstruction. Vis. Comput. 33(12), 1497–1509 (2017). https://doi.org/10.1007/s00371-016-1293-1

    Article  Google Scholar 

  28. Xu, K., Chen, K., Fu, H., Sun, W.L., Hu, S.M.: Sketch2Scene: sketch-based co-retrieval and co-placement of 3D models. ACM Trans. Graph. 32(4), 123:1–123:15 (2013). https://doi.org/10.1145/2461912.2461968

    Article  Google Scholar 

  29. Xu, K., Huang, H., Shi, Y., Li, H., Long, P., Caichen, J., Sun, W., Chen, B.: Autoscanning for coupled scene reconstruction and proactive object analysis. ACM Trans. Graph. 34(6), 177:1–177:14 (2015). https://doi.org/10.1145/2816795.2818075

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank YantingLin and Jian Wu. They helped with data preparation for our experiments. We would also like to thank the National Natural Science Foundation for their funding. This study was funded by the National Natural Science Foundation of China under Nos. 61472377, 61632006, and 6133101.

Author information

Authors and Affiliations

  1. Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China

    Siyu Hu & Xuejin Chen

  2. Microsoft Research Asia, Beijing, China

    Xin Tong

Authors
  1. Siyu Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuejin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xuejin Chen.

Ethics declarations

Conflict of interest

Siyu Hu declares that he has no conflict of interest. Xuejin Chen has received research Grants from Microsoft and Huawei Technology Co. Ltd. Xuejin Chen had visited Leonidas Guibass Group in Stanford University during February 21 to August 20, 2017. Xin Tong is researcher of Microsoft. He is associate editor of ACM TOG and IEEE TVCG. He is also guest professor of University of Science and Technology of China and Tianjin University.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, S., Chen, X. & Tong, X. Point sets joint registration and co-segmentation. Vis Comput 35, 1841–1853 (2019). https://doi.org/10.1007/s00371-018-1578-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-018-1578-7

Keywords

Search

Navigation