EC-Net: An Edge-Aware Point Set Consolidation Network

  • Lequan YuEmail author
  • Xianzhi Li
  • Chi-Wing Fu
  • Daniel Cohen-Or
  • Pheng-Ann Heng
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11211)


Point clouds obtained from 3D scans are typically sparse, irregular, and noisy, and required to be consolidated. In this paper, we present the first deep learning based edge-aware technique to facilitate the consolidation of point clouds. We design our network to process points grouped in local patches, and train it to learn and help consolidate points, deliberately for edges. To achieve this, we formulate a regression component to simultaneously recover 3D point coordinates and point-to-edge distances from upsampled features, and an edge-aware joint loss function to directly minimize distances from output points to 3D meshes and to edges. Compared with previous neural network based works, our consolidation is edge-aware. During the synthesis, our network can attend to the detected sharp edges and enable more accurate 3D reconstructions. Also, we trained our network on virtual scanned point clouds, demonstrated the performance of our method on both synthetic and real point clouds, presented various surface reconstruction results, and showed how our method outperforms the state-of-the-arts.


Point cloud Learning Neural network Edge-aware 



We thank anonymous reviewers for the comments and suggestions. The work is supported by the Research Grants Council of the Hong Kong Special Administrative Region (Project no. GRF 14225616), the Shenzhen Science and Technology Program (No. JCYJ20170413162617606 and No. JCYJ20160429190300857), and the CUHK strategic recruitment fund.

Supplementary material

474212_1_En_24_MOESM1_ESM.pdf (19.3 mb)
Supplementary material 1 (pdf 19793 KB)


  1. 1.
    Alexa, M., Behr, J., Cohen-Or, D., Fleishman, S., Levin, D., Silva, C.T.: Computing and rendering point set surfaces. IEEE Trans. Vis. Comput. Graph. 9(1), 3–15 (2003)CrossRefGoogle Scholar
  2. 2.
    Gal, R., Shamir, A., Hassner, T., Pauly, M., Cohen-Or, D.: Surface reconstruction using local shape priors. In: Symposium on Geometry Processing. Number EPFL-CONF-149318, pp. 253–262 (2007)Google Scholar
  3. 3.
    Sung, M., Kim, V.G., Angst, R., Guibas, L.J.: Data-driven structural priors for shape completion. ACM Trans. Graph. (SIGGRAPH Asia) 34(6), 175:1–175:11 (2015)Google Scholar
  4. 4.
    Xu, K., Kim, V.G., Huang, Q., Kalogerakis, E.: Data-driven shape analysis and processing. Comput. Graph. Forum 36(1), 101–132 (2017)CrossRefGoogle Scholar
  5. 5.
    Remil, O., Xie, Q., Xie, X., Xu, K., Wang, J.: Surface reconstruction with data-driven exemplar priors. Comput. Aided Des. 88, 31–41 (2017)CrossRefGoogle Scholar
  6. 6.
    Guerrero, P., Kleiman, Y., Ovsjanikov, M., Mitra, N.J.: PCPNet: learning local shape properties from raw point clouds. arXiv preprint arXiv:1710.04954 (2017)
  7. 7.
    Yu, L., Li, X., Fu, C.W., Cohen-Or, D., Heng, P.A.: PU-Net: point cloud upsampling network. arXiv preprint arXiv:1801.06761 (2018)
  8. 8.
    Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: AtlasNet: a Papier-Mâché approach to learning 3D surface generation. In: IEEE CVPR, pp. 216–224 (2018)Google Scholar
  9. 9.
    Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: IEEE CVPR, pp. 77–85 (2017)Google Scholar
  10. 10.
    Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., Stuetzle, W.: Surface reconstruction from unorganized points. In: Proceedings of SIGGRAPH, pp. 71–78 (1992)CrossRefGoogle Scholar
  11. 11.
    Turk, G., Levoy, M.: Zippered polygon meshes from range images. In: Proceedings of SIGGRAPH, pp. 311–318 (1994)Google Scholar
  12. 12.
    Amenta, N., Bern, M., Kamvysselis, M.: A new Voronoi-based surface reconstruction algorithm. In: Proceedings of SIGGRAPH, pp. 415–422 (1998)Google Scholar
  13. 13.
    Berger, M., et al.: A survey of surface reconstruction from point clouds. In: Computer Graphics Forum, vol. 36, pp. 301–329. Wiley Online Library (2017)Google Scholar
  14. 14.
    Lipman, Y., Cohen-Or, D., Levin, D., Tal-Ezer, H.: Parameterization-free projection for geometry reconstruction. ACM Trans. Graph. (SIGGRAPH) 26(3), 22:1–22:5 (2007)CrossRefGoogle Scholar
  15. 15.
    Huang, H., Li, D., Zhang, H.R., Ascher, U., Cohen-Or, D.: Consolidation of unorganized point clouds for surface reconstruction. ACM Trans. Graph. (SIGGRAPH Asia) 28(5), 176:1–176:8 (2009)Google Scholar
  16. 16.
    Pauly, M., Keiser, R., Kobbelt, L.P., Gross, M.: Shape modeling with point-sampled geometry. ACM Trans. Graph. (SIGGRAPH) 22(3), 641–650 (2003)CrossRefGoogle Scholar
  17. 17.
    Guennebaud, G., Barthe, L., Paulin, M.: Real-time point cloud refinement. In: Symposium on Point Based Graphics, pp. 41–48 (2004)Google Scholar
  18. 18.
    Fleishman, S., Cohen-Or, D., Silva, C.T.: Robust moving least-squares fitting with sharp features. ACM Trans. Graph. (SIGGRAPH) 24(3), 544–552 (2005)CrossRefGoogle Scholar
  19. 19.
    Öztireli, A.C., Guennebaud, G., Gross, M.: Feature preserving point set surfaces based on non-linear kernel regression. Comput. Graph. Forum (Eurographics) 28(2), 493–501 (2009)CrossRefGoogle Scholar
  20. 20.
    Huang, H., Wu, S., Gong, M., Cohen-Or, D., Ascher, U., Zhang, H.R.: Edge-aware point set resampling. ACM Trans. Graph. 32(1), 9:1–9:12 (2013)CrossRefGoogle Scholar
  21. 21.
    Preiner, R., Mattausch, O., Arikan, M., Pajarola, R., Wimmer, M.: Continuous projection for fast L1 reconstruction. ACM Trans. Graph. (SIGGRAPH) 33(4), 47:1–47:13 (2014)CrossRefGoogle Scholar
  22. 22.
    Guo, K., Zou, D., Chen, X.: 3D mesh labeling via deep convolutional neural networks. ACM Trans. Graph. 35(1), 3:1–3:12 (2015)CrossRefGoogle Scholar
  23. 23.
    Boulch, A., Marlet, R.: Deep learning for robust normal estimation in unstructured point clouds. Comput. Graph. Forum (SGP) 35(5), 281–290 (2016)CrossRefGoogle Scholar
  24. 24.
    Qi, C.R., Su, H., Niessner, M., Dai, A., Yan, M., Guibas, L.J.: Volumetric and multi-view CNNs for object classification on 3D data. In: IEEE CVPR, pp. 5648–5656 (2016)Google Scholar
  25. 25.
    Dai, A., Qi, C.R., Nießner, M.: Shape completion using 3D-encoder-predictor CNNs and shape synthesis. In: IEEE CVPR, pp. 5868–5877 (2017)Google Scholar
  26. 26.
    Han, X., Li, Z., Huang, H., Kalogerakis, E., Yu, Y.: High-resolution shape completion using deep neural networks for global structure and local geometry inference. In: IEEE ICCV, pp. 85–93 (2017)Google Scholar
  27. 27.
    Wang, P., Liu, Y., Guo, Y., Sun, C., Tong, X.: O-CNN: Octree-based convolutional neural networks for 3D shape analysis. ACM Trans. Graph. 36(4), 1–11 (2017)Google Scholar
  28. 28.
    Riegler, G., Ulusoy, A.O., Geiger, A.: OctNet: learning deep 3D representations at high resolutions. In: IEEE CVPR, pp. 6620–6629 (2017)Google Scholar
  29. 29.
    Liu, F., et al.: 3DCNN-DQN-RNN: a deep reinforcement learning framework for semantic parsing of large-scale 3D point clouds. In: IEEE ICCV, pp. 5678–5687 (2017)Google Scholar
  30. 30.
    Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems, vol. 30, pp. 5105–5114 (2017)Google Scholar
  31. 31.
    Hua, B.-S., Tran, M.-K., Yeung, S.-K.: Pointwise convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  32. 32.
    Klokov, R., Lempitsky, V.: Escape from cells: deep Kd-Networks for the recognition of 3D point cloud models. In: IEEE ICCV, pp. 863–872 (2017)Google Scholar
  33. 33.
    Landrieu, L., Simonovsky, M.: Large-scale point cloud semantic segmentation with superpoint graphs. arXiv preprint arXiv:1711.09869 (2017)
  34. 34.
    Xu, D., Anguelov, D., Jain, A.: PointFusion: Deep sensor fusion for 3D bounding box estimation. arXiv preprint arXiv:1711.10871 (2017)
  35. 35.
    Wang, W., Yu, R., Huang, Q., Neumann, U.: SGPN: similarity group proposal network for 3D point cloud instance segmentation. arXiv preprint arXiv:1711.08588 (2017)
  36. 36.
    Yang, Y., Feng, C., Shen, Y., Tian, D.: FoldingNet: interpretable unsupervised learning on 3D point clouds. In: IEEE CVPR, pp. 206–215 (2018)Google Scholar
  37. 37.
    Qi, C.R., Liu, W., Wu, C., Su, H., Guibas, L.J.: Frustum PointNets for 3D object detection from RGB-D data. arXiv preprint arXiv:1711.08488 (2017)
  38. 38.
    Li, Y., Bu, R., Sun, M., Chen, B.: PointCNN. arXiv preprint arXiv:1801.07791 (2018)
  39. 39.
    Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph CNN for learning on point clouds. arXiv preprint arXiv:1801.07829 (2018)
  40. 40.
    Su, H., et al.: SPLATNet: sparse lattice networks for point cloud processing. In: IEEE CVPR, pp. 2530–2539 (2018)Google Scholar
  41. 41.
    Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3D object reconstruction from a single image. In: IEEE CVPR, pp. 2463–2471 (2017)Google Scholar
  42. 42.
    Lin, C.H., Kong, C., Lucey, S.: Learning efficient point cloud generation for dense 3D object reconstruction. In: AAAI (2018, to appear)Google Scholar
  43. 43.
    Chang, A.X., et al.: ShapeNet: an information-rich 3D model repository. arXiv preprint arXiv:1512.03012 (2015)
  44. 44.
    Eberly, D.: Distance between point and triangle in 3D (1999).
  45. 45.
    Eberly, D.: Distance between point and line, ray, or line segment (1999).
  46. 46.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  47. 47.
    Bernardini, F., Mittleman, J., Rushmeier, H., Silva, C., Taubin, G.: The ball-pivoting algorithm for surface reconstruction. IEEE Trans. Vis. Comput. Graph. 5(4), 349–359 (1999)CrossRefGoogle Scholar
  48. 48.
    Kazhdan, M., Hoppe, H.: Screened poisson surface reconstruction. ACM Trans. Graph. 32(3), 29:1–29:13 (2013)CrossRefGoogle Scholar
  49. 49.
    Zheng, Y., Fu, H., Au, O.K.C., Tai, C.L.: Bilateral normal filtering for mesh denoising. IEEE Trans. Vis. Comput. Graph. 17(10), 1521–1530 (2011)CrossRefGoogle Scholar
  50. 50.
    Berger, M., Levine, J.A., Nonato, L.G.: A benchmark for surface reconstruction. ACM Trans. Graph. 32(2), 20 (2013)CrossRefGoogle Scholar
  51. 51.
    Lu, X., Wu, S., Chen, H., Yeung, S.K., Chen, W., Zwicker, M.: GPF: GMM-inspired feature-preserving point set filtering. IEEE Trans. Vis. Comput. Graph. 24(8), 2315–2326 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.The Chinese University of Hong KongShatinHong Kong
  2. 2.Tel Aviv UniversityTel AvivIsrael
  3. 3.Shenzhen Key Laboratory of Virtual Reality and Human Interaction TechnologyShenzhen Institutes of Advanced Technology, Chinese Academy of SciencesShenzhenChina

Personalised recommendations