Abstract
When obtaining three-dimensional (3D) face point cloud data based on structured light, factors related to the environment, occlusion, and illumination intensity lead to holes in the collected data, which affect subsequent recognition. In this study, we propose a hole-filling method based on stereo-matching technology combined with a B-spline. The algorithm uses phase information acquired during raster projection to locate holes in the point cloud, simultaneously extracting boundary point cloud sets. By registering the face point cloud data using the stereo-matching algorithm and the data collected using the raster projection method, some supplementary information points can be obtained at the holes. The shape of the B-spline curve can then be roughly described by a few key points, and the control points are put into the hole area as key points for iterative calculation of surface reconstruction. Simulations using smooth ceramic cups and human face models showed that our model can accurately reproduce details and accurately restore complex shapes on the test surfaces. Simulation results indicated the robustness of the method, which is able to fill holes on complex areas such as the inner side of the nose without a prior model. This approach also effectively supplements the hole information, and the patched point cloud is closer to the original data. This method could be used across a wide range of applications requiring accurate facial recognition.
摘要
在基于结构光的三维人脸点云数据采集过程中, 由于环境、 遮挡以及光照强度等因素影响, 采集到的数据往往会出现孔洞区域, 从而影响后续识别效果. 本文提出一种采用立体匹配技术结合B样条的孔洞修补方法. 算法首先利用光栅投影过程中获取的相位信息定位点云中的孔洞区域, 同时提取边界点集. 然后将立体匹配算法获取的人脸点云数据同光栅投影法采集的数据进行配准, 在孔洞处选取初始修补控制点. 再利用B样条曲线形状可由少数关键点大致描述这一特性, 将控制点作为关键点放入孔洞区域进行曲面重建迭代计算. 仿真使用光滑陶瓷杯和人脸模型进行, 结果表明, 该算法能够准确再现被测物体表面的细节和复杂形状. 同时也说明所提方法具有强鲁棒性, 能够在完全无先验信息的情况下对物体复杂区域实现孔洞修补, 并且修补后的点云更加接近原始数据. 该方法可广泛应用于需要精确人脸识别的领域.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bendels GH, Schnabel R, Klein R, 2006. Detecting holes in point set surfaces. J WSCG, 14:89–96.
Black JA Jr., Gargesha M, Kahol K, et al., 2016. Framework for performance evaluation of face recognition algorithms. SPIE 4862:163–174. https://doi.org/10.1117/12.473032
Buchin K, van Kreveld M, Meijer H, et al., 2009. On planar supports for hypergraphs. Proc 17th Int Conf on Graph Drawing, p.345–356. https://doi.org/10.1007/978-3-642-11805-0_33
Carr JC, Beatson RK, Cherrie JB, et al., 2001. Reconstruction and representation of 3D objects with radial basis functions. Proc 28th Annual Conf on Computer Graphics and Interactive Techniques, p.67–76. https://doi.org/10.1145/383259.383266
Chen H, Ma SW, Nuechter A, 2016. Non-synchronous point cloud algorithm for 3D reconstruction based on laser scanning and SFM. Chin J Sci Instrum, 37(5):1148–1157 (in Chinese). https://doi.org/10.3969/j.issn.0254-3087.2016.05.024
Chui CK, Lai MJ, 2000. Filling polygonal holes using C1 cubic triangular spline patches. Comput Aided Geom Des, 17(4): 297–307. https://doi.org/10.1016/S0167-8396(00)00005-4
Floater MS, Reimers M, 2001. Meshless parameterization and surface reconstruction. Comput Aided Geom Des, 18(2): 77–92. https://doi.org/10.1016/S0167-8396(01)00013-9
Furukawa Y, Ponce J, 2010. Accurate, dense, and robust multiview stereopsis. IEEE Trans Patt Anal Mach Intell, 32(8): 1362–1376. https://doi.org/10.1109/TPAMI.2009.161
Gilani ZS, Mian A, 2016. Towards large-scale 3D face recognition. Proc Int Conf on Digital Image Computing: Techniques and Applications, p.1–8. https://doi.org/10.1109/DICTA.2016.7797090
Goesele M, Curless B, Seitz SM, 2006. Multi-view stereo revisited. Proc IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.2402–2409. https://doi.org/10.1109/CVPR.2006.199
Huang XS, Zhang J, Fan LX, et al., 2017. A systematic approach for cross-source point cloud registration by preserving macro and micro structures. IEEE Trans Image Process, 26(7):3261–3276. https://doi.org/10.1109/TIP.2017.2695888
Huang Y, Da FP, 2019. Registration algorithm for point cloud based on normalized cross-correlation. IEEE Access, 7: 137136–137146. https://doi.org/10.1109/ACCESS.2019.2942127
Huang Y, Da FP, Tao HJ, 2015. An automatic registration algorithm for point cloud based on feature extraction. Chin J Lasers, 42(3):308002 (in Chinese). https://doi.org/10.3788/CJL201542.0308002
Jeong Y, Bok Y, Kim JS, et al., 2011. Complementation of cameras and lasers for accurate 6D SLAM: from correspondences to bundle adjustment. Proc IEEE Int Conf on Robotics and Automation, p.3581–3588. https://doi.org/10.1109/ICRA.2011.5979568
Jun Y, 2005. A piecewise hole filling algorithm in reverse engineering. Comput-Aided Des, 37(2):263–270. https://doi.org/10.1016/j.cad.2004.06.012
Kurlin V, 2014. A fast and robust algorithm to count topologically persistent holes in noisy clouds. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.1458–1463. https://doi.org/10.1109/CVPR.2014.189
Liu K, Zhou CH, Wei SB, et al., 2014. Optimized stereo matching in binocular three-dimensional measurement system using structured light. Appl Opt, 53(26):6083–6090. https://doi.org/10.1364/AO.53.006083
Liu YJ, Wang M, Zhang HL, et al., 2016. Strategy of classification and repairing for hole of incomplete point clouds based on fuzzy inference. J Comput Theor Nanosci, 13(11): 8227–8233. https://doi.org/10.1166/jctn.2016.5961
Nguyen VS, Trinh TH, Tran MH, 2015. Hole boundary detection of a surface of 3D point clouds. Proc Int Conf on Advanced Computing and Applications, p.124–129. https://doi.org/10.1109/ACOMP.2015.12
Orriols X, Binefa X, 2003. Finding breaking curves in 3D surface. Proc 1st Iberian Conf on Pattern Recognition and Image Analysis, p.681–688. https://doi.org/10.1007/978-3-540-44871-6_79
O’Toole AJ, An XB, Dunlop J, et al., 2012. Comparing face recognition algorithms to humans on challenging tasks. ACM Trans Appl Percept, 9(4):16. https://doi.org/10.1145/2355598.2355599
Pan YH, 2019. On visual knowledge. Front Inform Technol Electron Eng, 20(8):1021–1025. https://doi.org/10.1631/FITEE.1910001
Pan YH, 2021. Miniaturized five fundamental issues about visual knowledge. Front Inform Technol Electron Eng, 22(5):615–618. https://doi.org/10.1631/FITEE.2040000
Panchetti M, Pernot JP, Véron P, 2010. Towards recovery of complex shapes in meshes using digital images for reverse engineering applications. Comput-Aided Des, 42(8):693–707. https://doi.org/10.1016/j.cad.2010.01.004
Pernot JP, Moraru G, Véron P, 2007. Repairing triangle meshes built from scanned point cloud. J Eng Des, 18(5): 459–473. https://doi.org/10.1080/09544820701403797
Quinsat Y, Lartigue C, 2015. Filling holes in digitized point cloud using a morphing-based approach to preserve volume characteristics. Int J Adv Manuf Technol, 81(1–4): 411–421. https://doi.org/10.1007/s00170-015-7185-0
Russell, Stuart J, Norvig P, 2010. Artificial intelligence: a modern approach. Appl Mech Mater, 263(5):2829–2833.
Schaffer M, Grosse M, Harendt B, et al., 2011. High-speed three-dimensional shape measurements of objects with laser speckles and acousto-optical deflection. Opt Lett, 36(16):3097–3099. https://doi.org/10.1364/OL.36.003097
Shi LM, Guo FS, Hu ZY, 2011. An improved PMVS through scene geometric information. Acta Autom Sin, 37(5):560–568 (in Chinese). https://doi.org/10.3724/SP.J.1004.2011.00560
Stone EE, Skubic M, 2015. Fall detection in homes of older adults using the Microsoft Kinect. IEEE J Biomed Health Inform, 19(1):290–301. https://doi.org/10.1109/JBHI.2014.2312180
Wang JN, Oliveira MM, 2007. Filling holes on locally smooth surfaces reconstructed from point clouds. Image Vis Comput, 25(1):103–113. https://doi.org/10.1016/j.imavis.2005.12.006
Zheng EL, Wu CC, 2015. Structure from motion using structureless resection. Proc IEEE Int Conf on Computer Vision, p.2075–2083. https://doi.org/10.1109/ICCV.2015.240
Author information
Authors and Affiliations
Contributions
Yuan HUANG designed the research, processed the data, and drafted the paper. Feipeng DA helped organize the paper. Yuan HUANG and Feipeng DA revised and finalized the paper.
Corresponding author
Ethics declarations
Yuan HUANG and Feipeng DA declare that they have no conflict of interest.
Additional information
Project supported by the National Natural Science Foundation of China (No. 61405034), the Special Project on Basic Research of Frontier Leading Technology of Jiangsu Province, China (No. BK20192004C), the Shenzhen Science and Technology Innovation Committee (No. JCYJ20180306174455080), and the Natural Science Foundation of Jiangsu Province, China (No. BK20181269)
Rights and permissions
About this article
Cite this article
Huang, Y., Da, F. Three-dimensional face point cloud hole-filling algorithm based on binocular stereo matching and a B-spline. Front Inform Technol Electron Eng 23, 398–408 (2022). https://doi.org/10.1631/FITEE.2000508
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2000508