Segmentation phase measuring deflectometry for measuring structured specular surfaces

Abstract

Accurate and fast three-dimensional (3D) measurement for industrial products/components designed to possess 3D structured shapes is a key driver for improved productivity. However, challenges for current techniques are considerable to measure structured specular surfaces. A technique named segmentation phase measuring deflectometry (SPMD) is proposed in this paper, which enables structured specular surfaces to be measured with high accuracy in one setup. Concept of segmentation in topology is introduced into phase measuring deflectometry, which separates a surface with complex structures into continuous segments. Each segment can be reconstructed based on gradient information to achieve good form accuracy, and all reconstructed segments can be fused into a whole 3D strucutred form result based on their absolute spatial positioning data. Here, we propose and discuss the principle of SPMD, a segmentation technique to separate a strucured surface into segments, a spatial positioning technique to obtain absolute position of the segments, and a data fusion strategy to fuse all reconstructed segments. Experimental results show SPMD can achieve nanometer level accuracy for form measurement of continuous segments by comparing with stylus profilometer, which is significantly higher than the accuracy of direct phase measuring deflectometry. Meanwhile, SPMD has micron level spatial positioning accuracy for structures by measuring two specular steps and comparing with coordinate measuring machine, which differentiates this technique from gradient-based phase measuring deflectometry that extends measurement capability from continuous specular surfaces to complex structured specular surfaces. Compared with the existing measurement techniques, SPMD significantly improved the convenience and ability to measure freeform and structured specular surfaces with the advantages of high measurement accuracy, fast measurement, and potential application for embedded measurement.

1 Introduction

Driven by the ever-increasing demand for function integration, performance enhancement, and light weight, more and more next-generation industrial products/components are designed to possess three-dimensional (3D) structured shapes (i.e., non-rotational symmetric and discontinuous), to be made of multi-materials (e.g., materials with different reflective characteristics) and/or to integrate different shapes/structures [1, 2]. Accurate, fast, and in-line metrology is a key element underpinning future autonomous manufacturing systems. However, it is still a challenging assignment for current metrologies to measure structured specular surface. An example is seen in a monolithic multi-mirror arrays (shown in Fig. 1) on the Mid-Infrared instrument (MIRI) Spectrometer Optics for the James Webb Space Telescope [3]. It comprises 21 individual spherical mirrors of 11-mm length and 1-mm width. The relative 3D spatial position error of all the individual mirrors requires < 20 µm. Meanwhile, form accuracy of individual mirror must be better than 50 nm root mean square (RMS) [4]. The positions of the centers of each mirror are varied in a space coordinate system, and the big steps between each of the mirrors make it difficult for any of the exit measurement instrument to perform a measurement of the whole part in one setup. Coordinate measuring machine (CMM) [5] and stylus profilometer [6] are measurement techniques that are well accepted in industry. CMM records detailed dimensional figures by using a probe moving along workpiece surface; therefore, it can be used to accurately measure a surface with complex geometric shape. Stylus profilometer uses a stylus tip to physically touch the measured surface and evaluates the surface error by fitting the collected data and comparing with ideal surface model. Measurement accuracy of stylus profilometer can be achieved to a few tens of nanometers. However, because both stylus profilometer and CMM are based on point scanning, these techniques are time-consuming and have a potential damage for the test surface by the stylus tip. Moreover, some disadvantages in stylus profilometer and CMM, such as high cost and large volume, are difficult to be solved due to requirements of their measurement principles. Interferometry is a well-known non-contact optical measurement method based on interference principle [7]. Shore et al. tried to test the multi-mirror arrays by using a Twyman Green phase shifting interferometer [4]. The interferometer can successfully obtain the form data of individual mirrors. However, it has to be mounted on a three-axis machine in order to measure the spatial positioning of the individual mirrors, which results in that the measurement procedures were very complex and the measurement accuracy requirements for the whole part were difficult to achieve.

Fig. 1

Picture of monolithic multi-mirror arrays

Phase measuring deflectometry (PMD) is an optical technique developed for 3D measurement of freeform specular surface with advantages of good accuracy, low cost, and full-field and non-contact measurement [8,9,10]. According to approaches of obtaining form information, PMD can be divided into two categories: gradient-based phase measuring deflectometry (GPMD) [11,12,13,14], and direct phase measuring deflectometry (DPMD) [15, 16]. GPMD obtains gradients of the measured surface using phase data and then reconstructs form data based on gradient integration. In contrast, DPMD directly calculates form information from phase data instead of gradients by establishing the relationship between phase value and depth change based on mathematical model. GPMD can generally achieve submicron measurement accuracy or even tens of nanometers accuracy in the case of measuring low curvature surfaces [17], because of high sensitivity of gradient variation. Zhang et al. proposed a deflectometry method based on 3D digital image correlation for measuring the form of specular surface [18]. A liquid crystal display (LCD) screen is required to be moved during a measurement process; therefore, this system is not suitable for embedded measurement since the movement of screen needs large space. GPMD can only measure continuous surface due to the limitation of the principle of integral method. By contrast, DPMD can measure discontinuous specular surface because form data are extracted from phase value directly. Shi et al. developed an infrared PMD which combines a binary fringe defocusing technique and DPMD [19]. However, significant phase noise is introduced to the calculation process of this technique, which results in measurement accuracy of DPMD to only reach tens of microns [15, 16]. A measurement of the multi-mirror arrays mentioned above has been studied by using DPMD [15]. Though rough form data of the whole part can be obtained successfully, measurement accuracy of DPMD is far from meeting the requirement of individual mirror. Wang et al. studied a method to improve measurement accuracy of DPMD by introducing stereo algorithm [20]; however, DPMD is difficult to be applied in embedded measurements since two screens used in the system result in large system volume. We classify stereo deflectometry [21, 22] as a kind of GPMD technique. Note that stereo deflectometry can obtain absolute depth data in the process of solving gradient information, while other GPMD techniques can only obtain gradient data. Therefore, in theory, stereo deflectometry has a capacity of reconstructing a structured specular surface based on the obtained absolute depth data. However, because the absolute depth accuracy is not good (almost in the same order of magnitude as DPMD) compared with the accuracy of the depth reconstructed with gradients, scholars generally ignore the directly obtained absolute depth information and focus on using its ability of gradient measurement to detect continuous specular surfaces [17, 21, 22]. Overall, challenges for current techniques are considerable when measuring structured specular surfaces represented by the multi-mirror arrays, requiring important innovations in measurement techniques to achieve the ambitious target of accurate, embedded, and in-line measurement. Here, we propose and demonstrate a method to enhance the measurement capability of PMD, which we call segmentation phase measuring deflectometry (SPMD). Three novel techniques are presented in this paper: segmentation technique, spatial positioning technique, and data fusion strategy. Segmentation, the concept that we first applied in terms of PMD, working together with our developed spatial positioning technique and data fusion strategy, enables SPMD to perform a measurement of structured specular surfaces with high accuracy in one setup. The concept of segmentation in SPMD means separating a surface with complex structures into several continuous segments. The developed spatial positioning technique can establish the relative locations of the segments with micron level accuracy. Individual segment is reconstructed and smoothed based on gradient data, which enables form accuracy of individual segment to approach nanometer level accuracy. Data fusion strategy is studied to connect individual segment with the spatial location. The concept of this technique promises remarkable advancement of 3D measurement of structured specular surfaces with its advantages of high accuracy, fast speed, and full-field measurement.

2 Principle and methods

2.1 Introduction of SPMD

A SPMD system consists of a fringe displaying screen and two imaging sensors, whose working principle is illustrated in Fig. 2. The screen, the structured specular surface under test, and individual imaging sensor form a spatial triangle, so that imaging sensors can detect the fringes on the screen through the reflection of the specular surface. Compared with the fringe displayed on the screen, the fringe captured by imaging sensors is deformed due to the shape of the measured surface. Processing the deformed fringes on the imaging sensors with our developed spatial positioning method, spatial location and gradients of the surface can be obtained simultaneously. During a measurement process, the screen is required to display multiple phase-shifting sinusoidal fringe patterns in vertical and horizontal directions. A two-dimensional (2D) phase coordinate system can be created by extracting mutually perpendicular absolute phase values from the displayed fringe patterns. Since a well-manufactured LCD screen is generally accepted as the fringe displaying screen in current PMD system, it is quite simple to establish the relationship between the phase coordinate system and a 2D physical coordinate system on the screen based on the screen pixel size, so that the physical location on the screen corresponding to a phase coordinate can be calculated accurately. When the fringe patterns are displayed on the screen, the imaging sensors synchronously capture each reflected fringe pattern. Two mutually perpendicular absolute phase maps carrying the form information of the measured specular surface can be extracted from the captured fringe patterns by applying phase shifting and phase unwrapping algorithms. Individual phase map is a 3D matrix, which consists of pixel coordinates of the imaging sensor as horizontal and vertical coordinates and phase values as depth coordinate. Therefore, if knowing a pixel coordinate, the corresponding coordinate in terms of the phase coordinate system can be obtained by applying 2D interpolating to horizontal absolute phase map and vertical absolute phase map, using $S$ to represent an arbitrary space point near the measured surface. Its image point, denoted with ${P}_{1}$, in terms of the pixel coordinate system of an imaging sensor can be easily obtained after calibrating of the imaging parameters of the sensor. Phase coordinate of ${P}_{1}$ in terms of the phase coordinate system can be obtained by applying the interpolation method based on the pixel coordinate of ${P}_{1}$. Then, the corresponding physical location on the screen, denoted with ${Q}_{1}$, can be calculated based on the relationship between the phase coordinate system and the physical coordinate system. Assuming $S$ is a point on the measured surface, incident light ${Q}_{1}S$ and outgoing light $S{P}_{1}$ should obey the law of reflection. Consequently, an equivalent normal vector of $S$ can be calculated based on ${Q}_{1}$, $S$, and ${P}_{1}$, which is expressed by $\stackrel{\rightharpoonup}{{{\varvec{n}}}_{1}}$. Similarly, a new equivalent normal vector of $S$ can be obtained based on its image point ${P}_{2}$ on another imaging sensor, which is denoted with $\stackrel{\rightharpoonup}{{{\varvec{n}}}_{2}}$. As shown in Fig. 2, only when $S$ in on the surface, $\stackrel{\rightharpoonup}{{{\varvec{n}}}_{1}}$ is equal to $\stackrel{\rightharpoonup}{{{\varvec{n}}}_{2}}$, and both are equal to the real normal vector of the measured specular surface. Therefore, we can change the spatial coordinate of $S$, e.g. by changing the coordinate of $S$ along a camera ray, to obtain a position (denoted as ${S}_{1}$) where the difference between $\stackrel{\rightharpoonup}{{{\varvec{n}}}_{1}}$ and $\stackrel{\rightharpoonup}{{{\varvec{n}}}_{2}}$ reaches minimum. Here, we use $\left(\begin{array}{ccc}{x}_{\left(\mathrm{1,1}\right)}& {y}_{\left(\mathrm{1,1}\right)}& {z}_{\left(\mathrm{1,1}\right)}\end{array}\right)$ and $\left(\begin{array}{ccc}{n}_{{x}_{\left(\mathrm{1,1}\right)}}& {n}_{{y}_{\left(\mathrm{1,1}\right)}}& 1\end{array}\right)$ to express the spatial coordinate and the calculated normal vector of ${S}_{1}$, respectively. Note that though this process is similar to the gradient obtain process of stereo deflectometry, the difference is that the calculated absolute depth data is commonly disregarded in stereo deflectometry, while in SPMD, both spatial positioning data and gradient data are fully utilized based on the concept of segmentation to participate in the later data fusion process.

Fig. 2

Illustration of the principle of SPMD. Green ball stands for a space point. Yellow arrow and purple arrow show equivalent normal vector calculated based on two imaging sensors, respectively. Red arrow represents normal vector of the measured surface

Phase error is inevitably involved in the process of extracting absolute phase values from the captured fringe patterns, which is resulted by complex factors such as electronic noise of the screen and imaging sensors, nonlinear error in fringe display and capture, fringe blur caused by lens defocus, and calibration error etc., using $\delta Q$ to represent physical location error on the screen caused by the phase error. With the benefit of our previous study [23], the influence of $\delta Q$ on spatial positioning accuracy can be expressed with Eq. (1), regardless of calibration error in terms of the relative position between the screen and the imaging sensors:

$$\delta z=\frac{tan\gamma }{1+{L}_{s}/{L}_{d}}\delta Q$$

(1)

where $\delta z$ represents spatial positioning error in depth direction.${L}_{s}$ and ${L}_{d}$ represent the length from the measured surface to the screen and to the imaging sensor, respectively. $\gamma$ denotes the angle between surface normal and camera ray. In a SPMD system, $\gamma$ is around 45°. Assuming ${L}_{d}$ is $k$ times of ${L}_{s}$ ($\mathrm{k}$ is generally changing from 1 to 4 in a SPMD system), Eq. (1) can be simplified as Eq. (2).

$$\delta z=\frac{k}{1+k}\delta Q$$

(2)

According to Eq. (2), $\delta z$ and $\delta Q$ are in the same order of magnitude (which is tens of micrometers in our experiments). Though the result of spatial positioning can present 3D information of the measured surface in terms of micrometer scale, the accuracy requires further improvement for measuring a surface with nanometer level accuracy requirement. To realize this purpose, gradient data are studied to optimize and smooth the measurement data. Our previous work [23] shows that the relationship between $\delta Q$ and gradient error can be described as Eq. (3).

$$\delta \alpha =\frac{1}{2{L}_{s}}\delta Q$$

(3)

where $\delta \alpha$ is the gradient error caused by $\delta Q$. According to Eqs. (2) and (3), gradient data is ~$2{L}_{s}k/(k+1)$ times better than spatial positioning data in terms of restraining the influence of phase error. ${L}_{s}$ is typically larger than 100 mm in a PMD system, which leads to reconstructed surface based on gradient integration to achieve submicron or even tens of nanometers form accuracy. However, an obvious defect of gradient integration technique is that it can only be applied to a continuous surface. Serious measurement error will be introduced when reconstructing a structured surface with gradient data. Here, we use a concept of segmentation to solve this problem as shown in Fig. 2. The measured structured specular surface is partitioned into multiple continuous segments with gradient information. Individual continuous segment can be reconstructed with their gradient data to achieve nanometer level form accuracy. However, the reconstruction process results in a missing of the absolute depth information of the reconstructed segment. A data fusion technique must be conducted to recover the absolute position in space of the reconstructed segment to reconstruct the 3D form of the whole structured specular surface.

2.2 Spatial positioning and segmentation process in SPMD

Figure 3 demonstrates the process of obtaining absolute 3D coordinates and gradients information of a structured specular surface in SPMD. Calibration is an important and first step of the process, which is to obtain the transformation matrix ${{\varvec{H}}}_{{\varvec{L}}}$ between screen coordinate system $\{L\}$ and phase coordinate system, the relationship between $\{L\}$ and camera coordinate systems ($\{{C}_{1}\}$ and $\{{C}_{2}\}$), transformation matrix ${{\varvec{H}}}_{{\varvec{c}}}$ between $\{{C}_{1}\}$ and $\{{C}_{2}\}$, imaging matrix ${{\varvec{A}}}_{1}$ from $\left\{{C}_{1}\right\}$ to image coordinate system $\{{P}_{1}\}$ of imaging sensor 1, and imaging matrix ${{\varvec{A}}}_{2}$ from $\left\{{C}_{2}\right\}$ to image coordinate system $\{{P}_{2}\}$ of imaging sensor 2. For an image point ${m}_{(i,j)}^{1}$ ($i$ and $j$ represent coordinate in $u$ and $v$ direction of pixel map) in imaging sensor 1, a unique physical point in screen point ${M}_{(i,j)}^{1}$ can be obtained based on ${{\varvec{H}}}_{{\varvec{L}}}$ and phase values corresponding to ${m}_{(i,j)}^{1}$. An equivalent normal ${\overrightarrow{n}}_{{(i,j)}_{k}}^{1}$ can be calculated based on ${m}_{(i,j)}^{1}$, ${M}_{(i,j)}^{1}$ and a point ${S}_{{(i,j)}_{k}}^{1}$ ($k$ denotes index number) on extended line of ${{m}_{(i,j)}^{1}O}_{1}$ (${O}_{1}$ is the optical center of imaging sensor 1), using ${S}_{{(i,j)}_{k}}^{2}$ and ${\overrightarrow{n}}_{{(i,j)}_{k}}^{2}$ to represent the values of ${S}_{{(i,j)}_{k}}^{1}$ and ${\overrightarrow{n}}_{{(i,j)}_{k}}^{1}$ expressed in terms of $\{{C}_{2}\}$, which can be obtained based on ${{\varvec{H}}}_{{\varvec{c}}}$. Imaging point ${m}_{(i,j)}^{2}$ in $\{{C}_{2}\}$ of ${S}_{{(i,j)}_{k}}^{2}$ can be calculated with imaging matrix ${{\varvec{A}}}_{2}$. Assuming ${S}_{{(i,j)}_{k}}^{2}$ is a reflection point with normal ${\overrightarrow{n}}_{{(i,j)}_{k}}^{2}$, luminous point of ${m}_{(i,j)}^{2}$ can be obtained according to the law of reflection, which is denoted with ${\widehat{M}}_{(i,j)}^{2}$. Using ${d}_{(i,j)}$ to express the difference between ${\widehat{M}}_{(i,j)}^{2}$ and the corresponding point ${M}_{(i,j)}^{2}$ on the screen of ${m}_{(i,j)}^{2}$, a true reflection point can be obtained by searching points along extended line of ${{m}_{(i,j)}^{1}O}_{1}$ by minimizing ${d}_{(i,j)}$. Repeating the search process along imaging ray of every camera pixels, we can obtain a $m\times n$ spatial coordinates matrix $\left[\begin{array}{ccc}\left(\begin{array}{ccc}{x}_{\left(\mathrm{1,1}\right)}& {y}_{\left(\mathrm{1,1}\right)}& {z}_{\left(\mathrm{1,1}\right)}\end{array}\right)& \cdots & \left(\begin{array}{ccc}{x}_{\left(1,n\right)}& {y}_{\left(1,n\right)}& {z}_{\left(1,n\right)}\end{array}\right)\\ \vdots & \ddots & \vdots \\ \left(\begin{array}{ccc}{x}_{\left(m,1\right)}& {y}_{\left(m,1\right)}& {z}_{\left(m,1\right)}\end{array}\right)& \cdots & \left(\begin{array}{ccc}{x}_{\left(m,n\right)}& {y}_{\left(m,n\right)}& {z}_{\left(m,n\right)}\end{array}\right)\end{array}\right]$ where $m$ and $n$ represent the number of imaging pixels in horizontal and vertical directions, respectively. The coordinates matrix records the spatial positioning of the measured structured specular surface. Meanwhile, a normal vector matrix $\left[\begin{array}{ccc}\left(\begin{array}{ccc}{n}_{{x}_{\left(\mathrm{1,1}\right)}}& {n}_{{y}_{\left(\mathrm{1,1}\right)}}& 1\end{array}\right)& \cdots & \left(\begin{array}{ccc}{n}_{{x}_{\left(1,n\right)}}& {n}_{{y}_{\left(1,n\right)}}& 1\end{array}\right)\\ \vdots & \ddots & \vdots \\ \left(\begin{array}{ccc}{n}_{{x}_{\left(m,1\right)}}& {n}_{{y}_{\left(m,1\right)}}& 1\end{array}\right)& \cdots & \left(\begin{array}{ccc}{n}_{{x}_{\left(m,n\right)}}& {n}_{{y}_{\left(m,n\right)}}& 1\end{array}\right)\end{array}\right]$ can be obtained, which records the gradient information of the measured surface. Note that even for a true reflection point, ${d}_{(i,j)}$ cannot equal zero due to calibration and phase noise error. However, ${d}_{(i,j)}$ of a true reflection point (point on the surface of the measured structured specular mirror) is still much small compared with those obtained by points beyond the sample under test (SUT) and points on the edge of segments. Therefore, a threshold can be used to remove non-surface points and edge points in spatial coordinates matrix and normal vector matrix by evaluating the size of ${d}_{(i,j)}$. In our experiments, the threshold is commonly at micron level for a well-calibrated system. The monolithic multi-mirror arrays on the MIRI Spectrometer Optics shown in Fig. 1 are used as measurement sample to demonstrate the segmentation process. Figure 4a and b show the obtained matrixes of normal vector along x and y direction, where non-surface points and edge points have been removed based on the threshold method. Figure 4c shows a two direction (2D) projection in z direction of a normal vector matrix expressed in terms of pixel coordinate system. Each continuous segment of the normal vector matrix can be clearly distinguished after the removement with the threshold method, as shown in Fig. 4c. In order to achieve the segmentation, a region of interest (ROI) covering a whole target segment is selected, as shown in Fig. 4c. The data outside the target segment may be covered by the ROI especially when a target segment has complex shape. In order to eliminate the data outside target segment (DOTS) in ROI, a region mask can be obtained by binarizing the selected ROI, as shown in Fig. 4d. Then, a threshold can be set up according to the pixel number of the target segment to eliminate the region of DOTS which has less pixel number than the threshold. Figure 4e is the result after the elimination of the region of DOTS, which is used as a region mask to accurately extract the target segment. All continuous segments of the measured sample can be extracted by repeating segmentation process. A flowchart is used to summarize the segmentation process of SPMD, as shown in Fig. 5.

Fig. 3

Illustration of the spatial positioning of SPMD

Fig. 4

Illustration of the segmentation process of SPMD. a Matrix of normal vector along x direction. b Matrix of normal vector along y direction. c Z-direction value of normal vector matrix expressed in pixel coordinate system. d Region mask of ROI. e Region mask after removing ROS

Fig. 5

Summary of segmentation process of SPMD

2.3 Data fusion

Figure 6 is used to illustrate the data fusion strategy of SPMD by measuring the monolithic multi-mirror arrays. Two imaging sensors capture the deformed fringe patterns along x and y directions through the reflection of the measured mirror. By processing the captured patterns with the developed spatial positioning technique, rough space point cloud with spatial positioning information and gradient data along x, y direction of the mirror can be obtained. Then, a target continuous segment is separated from the structured mirror by extracting the corresponding region mask in terms of pixel coordinates based on the proposed segmentation method. Data in depth direction of the target segment can be reconstructed based on the gradient data and x, y coordinates. On one hand, the reconstructed depth data improves form accuracy of the target segment compared with the rough 3D position data, as shown in Fig. 6. On the other hand, the reconstructed depth data is a relative value; therefore, a method is required to be studied to recapture the absolute spatial position of the reconstructed target segment to obtain the spatial location relationship between segments, using matrices $M$ and $\widehat{M}$ to represent spatial coordinates of a point on target segment and the corresponding point on the reconstructed segment with the same horizontal and vertical coordinates, respectively. $M$ is denoted with $\left(\begin{array}{ccc}x& y& z\end{array}\right)$ and $\widehat{M}$ is expressed by $\left(\begin{array}{ccc}x& y& \widehat{z}\end{array}\right)$. The relationship between $M$ and $\widehat{M}$ can be expressed with Eq. (4).

$$M={\varvec{R}}\widehat{M}+{\varvec{T}}+e$$

(4)

where ${\varvec{R}}$ is a 3 × 3 identity matrix since there is no rotation transformation of data in the process of reconstruction. ${\varvec{T}}$ equals $\left[\begin{array}{ccc}0& 0& {d}_{z}\end{array}\right]$, because $M$ and $\widehat{M}$ have the same coordinates in x and y direction. $e$ represents measurement noise in spatial positioning data of rough space point cloud. ${d}_{z}$ can be calculated by minimizing the following functional with least square method:

$$\sum\nolimits_{i=1}^{m}{\left|{M}_{i}-({\varvec{R}}{\widehat{M}}_{i}+{\varvec{T}})\right|}^{2}$$

(5)

where $i$ is the index of sampling point. $m$ is the number of sampling points on the reconstructed segment. Similarly, all continuous segments can be reconstructed with gradient data, and recaptured absolute depth data based on the fusion technique. So far, improvement of measurement accuracy of the structured mirrors has completed. Final step is to establish the relationship between the 21 reconstructed structured mirrors and the base of the mirror to complete the reconstruction of the whole structured surface. This step can be achieved through a fusion process by using the reconstructed mirrors to replace the corresponding rough 3D position data. A comparison between final fusion result and rough position data shown in Fig. 6 can reflect the improvement of surface accuracy after the fusion process.

Fig. 6

Diagram of data fusion strategy

3 Experiments and results

We measured three structured specular mirrors to evaluate the measurement accuracy of SPMD in terms of spatial positioning accuracy and form measurement accuracy to an individual continuous segment, respectively. The SPMD system and the sample under test (SUT) are shown in Fig. 7a. A tablet (iPad Pro 12.9-inch, Apple) is used as the fringe-displaying screen of the SPMD system in experiments. Two CCD (Charge-coupled Device) cameras (Lw235M, Lumenera) are used as the imaging sensors in the system. Resolution of the cameras is 1616 × 1216 pixels. 35 mm fixed focal length lens (MVL35M23, Navitar) is mounted on the cameras. The cameras focus on the SUT during the measurement process. Fringe patterns are generated by programming with Matlab. Two groups of perpendicular sinusoidal fringe patterns with fringe numbers of 225, 224, and 210 are generated in our experiments according to requirements of phase calculation algorithms. Duet Display is used to turn the tablet into a second monitor to sequentially display the generated fringe patterns with high performance. A sinusoidal fringe period of the fringe pattern with fringe number of 225 takes up 10 screen pixels. The displayed fringe patterns are captured by the cameras through the reflection of the SUT. In order to reduce phase noise, we averaged of every 5 obtained fringe images. Then, eight-step phase-shifting algorithm is used to calculate wrapped phase map with the averaged fringe images. Absolute phase map can be obtained by unwrapping the wrapped phase map with optimized 3-frequency selection method [24, 25]. The SPMD system in experiments is calibrated based on holistic calibration method [22]. During the calibration process, a high-precision plane mirror moves arbitrarily several times around the position of the object to be measured. The screen in SPMD is reflected by the plane mirror and captured by the imaging sensors. Therefore, the imaged screen can be used as a phase target to calibrate the imaging parameter of the imaging sensors [26, 27]. Then, the transformation relationship between each imaged screen and an imaging sensor can be used to calculate the spatial relationship between the real screen and an imaging sensor in SPMD. Finally, an iterative optimization is applied to optimize the calibration parameters in SPMD. Least-squares method [28] is applied to reconstruct form data of a continuous segment with gradient data. The principle of the reconstruction method is based on an approximation that the normal vector of a point is perpendicular to the vectors connecting points at either side. Radial basis function integration (RBFI) method [29] is used as a supplementary constraint of the reconstruction method to integrate a small measurement data set during the reconstruction process.

Fig. 7

SPMD system and samples. a Pictures of SPMD system and three samples under test in our experiments. b A picture of the captured rectangular step gauge. c A picture of the captured circular step gauge. d A picture of the captured concave mirror with four small concave features

3.1 Evaluation of spatial positioning accuracy

Sample M1 shown in Fig. 7a is a rectangular step gauge. A picture of fringe pattern through the reflection of M1 is shown in Fig. 7b. The form data of the step gauge measured by SPMD is shown in Fig. 8a. Meanwhile, we used a CMM (ZEISS PRISMO) to measure the step gauge to compare with the results of SPMD. Figure 8b shows the measurement difference between the two techniques, using ${h}_{21}$, ${h}_{31}$, and ${h}_{41}$ to represent the distance between the second step and the first step, the distance between the third step and the first step, and the distance between the fourth step and the first step respectively, as shown in Fig. 7a. Table 1 shows a comparison between SPMD and CMM by measuring sample M1. Step distances ${{\varvec{h}}}_{\varvec{21}}$, ${{\varvec{h}}}_{\varvec{31}}$, and ${{\varvec{h}}}_{\varvec{41}}$ measured by CMM are 2994.6 µm, 6921.0 µm, and 11863.0 µm, respectively. The step distances measured by SPMD are 3011.7 µm, 6928.9 µm, and 11852.8 µm, respectively. The differences between the two techniques in the measurement of ${{\varvec{h}}}_{\varvec{21}}$, ${{\varvec{h}}}_{\varvec{31}}$, and ${{\varvec{h}}}_{\varvec{41}}$ are 17.1 µm, 7.9 µm, and −10.2 µm, respectively. Sample M2 shown in Fig. 7a is a specular mirror with three circular steps. Figure 7c is a picture of the sample with deformed fringe information. Figure 8c shows measurement result of the sample by SPMD. Measurement difference of the circular step gauge by SPMD and CMM is shown in Fig. 8d. The distances from three circular steps to the circular base of the mirror are denoted as ${h}_{1}$, ${h}_{2}$, and ${h}_{3}$ respectively, as shown in Fig. 7a. Table 2 shows a comparison between SPMD and CMM by measuring sample M2. Step distances ${{\varvec{h}}}_{\varvec{1}}$, ${{\varvec{h}}}_{\varvec{2}}$, and ${{\varvec{h}}}_{\varvec{3}}$ measured by CMM are 5617.0 µm, 4671.8 µm, and 3675.6 µm, respectively. Results obtained by SPMD are 5626.5 µm, 4660.7 µm, and 3682.1 µm, which indicates that the differences between SPMD and CMM in measuring the step distances are 9.5 µm, −11.1 µm, and 6.5 µm, respectively. Measurement results in Tables 1 and 2 show that the spatial positioning accuracy of SPMD is below 20 µm, which reflects the measurement accuracy of specular structure positions.

Fig. 8

Experimental evaluation of spatial positioning accuracy of SPMD. a Spatial positioning data of the rectangular step gauge measured by SPMD. b Difference between SPMD and CMM in the measurement of the rectangular step gauge. c Spatial positioning data of the specular mirror with three circular steps measured by SPMD. d Difference between SPMD and CMM in the measurement of the circular step gauge

Table 1 Comparison between SPMD and CMM by measuring sample M1 (unit: µm)

Table 2 Comparison between SPMD and CMM by measuring sample M2 (unit: µm)

3.2 Evaluation of form measurement accuracy to an individual continuous segment

Sample M3 shown in Fig. 7a is a concave specular mirror with four small concave features. Figure 7d shows a captured picture of the sample with fringe information. Figure 9a shows the spatial positioning data of a concave feature (enlarged in Fig. 7d) obtained by SPMD. Significant measurement noises can be seen clearly in Fig. 9a. Form information of the concave feature is hidden by these noises. Then, the concave feature is separated from the whole part as a continuous segment and reconstructed with gradient data. The reconstructed result is shown in Fig. 9d, which clearly displays the form of the concave feature. Figure 9b shows the spatial positioning data of the mirror part within region of interest (ROI) obtained by SPMD, where it is difficult to distinguish the small concave features due to measurement noise in the spatial positioning data. By contrast, the final measurement result of the mirror part within ROI after data fusion is shown in Fig. 9e, which can clearly display the form of the whole mirror. A stylus profilometer (Formtracer CS-5000CNC, Mitutoyo) with 70 nm measurement accuracy is used to evaluate the form measurement accuracy of SPMD to an individual continuous segment. An ideal concave is fitted based on measurement result of the stylus profilometer. Root mean square (RMS) of the difference between the spatial positioning data of SPMD and the fitted concave is 34.7 µm, as shown in Fig. 9c. In contract, Fig. 9f shows that RMS of the difference between the reconstructed data of SPMD and the fitted concave is 39.1 nm.

Fig. 9

Experimental evaluation of form measurement accuracy of SPMD to an individual continuous segment. a Spatial positioning data of the concave feature obtained by SPMD. b Spatial positioning data of the mirror part within ROI obtained by SPMD. c Difference between spatial positioning data of SPMD and fitted results of stylus profilometer in the measurement of the concave feature. d Reconstruction data of the concave feature obtained by SPMD. e Final measurement result of the mirror part within ROI obtained by SPMD. f Difference between reconstruction data of SPMD and fitted results of stylus profilometer in the measurement of the concave feature

4 Discussion

The conceptual essence of SPMD is not only the concept of segmentation but also the implementation of the “spatial positioning” and “fusion strategy,” which enables SPMD to collect both absolute spatial positioning data and gradient data, and integrate these two kinds of data into a whole 3D data. This solves the measurement limitation of current PMD techniques while maintaining their advantages. Compared with GPMD, SPMD extends the measurement capability from continuous specular surfaces only to complex structured specular surfaces. Meanwhile SPMD provides much higher measurement accuracy compared with DPMD. In the experiment of measuring a concave specular mirror with four small concave features, we compared the spatial positioning data obtained by SPMD and the reconstruction data based on gradient data. Since the spatial positioning accuracy of SPMD is in the same order of magnitude as the measurement accuracy of DPMD, the comparison can reflect the difference between SPMD and DPMD. The experimental results show that SPMD’s spatial positioning accuracy and form measurement accuracy to an individual continuous segment are tens of micrometers and tens of nanometers respectively, which indicates the accuracy of SPMD is nearly 1000 times higher than that of DPMD in a measurement of continuous specular segments. This shows that SPMD has obvious advantages over DPMD in the detection ability of a specular surface with small curvature change. Like traditional PMD system, it is important to consider two main factors that affect the measurement accuracy of SPMD. One is calibration accuracy, which includes the calculation accuracy of imaging model parameters, the relationship between screen pixel coordinates and physical position coordinates, and the spatial position relationship between cameras and screen. Although scholars have developed several approaches to enhance the calibration accuracy of PMD [17, 22, 30], further studies are still required to improve measurement accuracy of SPMD. Besides calibration accuracy, phase error is another factor. This relates to a problem that we must consider in the process of measurement — phase error caused by curvature of measured surface. Because the curvature of a measured surface will blur the collected fringe pattern to a certain extent, which results in the increase of phase error, the accuracy of SPMD could be relatively reduced when measuring a specular surface with large curvature change. One way to solve this problem is to increase the depth of field (DOF) of the imaging sensors. In short, we believe that with the improvement of calibration accuracy and phase accuracy of SPMD in the future, its spatial positioning accuracy and form measurement accuracy to an individual continuous segment will be further improved.

The measurement speed of SPMD depends on the number of fringe images needed for a complete measurement and the interval time between adjacent image captures. The applied phase calculation methods determine how many fringe images are needed. The interval time between adjacent pattern captures is affected by the display speed of the screen and the capture speed of the imaging sensor. In our measurement, an imaging sensor takes 240 images (including 120 horizontal fringe patterns and 120 vertical fringe patterns) for a complete measurement. In fact, the image number can be reduced according to measurement requirements (e.g., single-shot PMD has been studied in literature [11, 31]). The imaging sensor used in the experiment has a speed of 12 frames per second. Theoretically our system needs 20 s to complete the acquisition of 240 fringe images according to the imaging sensor speed. However, because the screen has a relative low display speed, the imaging sensors have to spend plenty of time waiting for the screen to display a fringe pattern. The resulting acquisition time for one measurement is around 1 min. However, we believe that it is not a tough challenge to improve the measurement speed of SPMD. Acquisition time for one measurement could achieve around 1 s or possibly even faster in the future if high-speed imaging sensor and screen are applied, and synchronization between them is realized. Since measurement speed of SPMD can be fast, there is no need to cooperate with heavy experimental platform that required by traditional measurement system (e.g., CMM and stylus profilometer) to reduce the impact of vibration. This increases systematic flexibility so that SPMD can be designed to a portable measurement system.

Compared with the existing measurement methods in industry (e.g., CMM, stylus profilometer, and interferometry), our developed techniques differentiate SPMD from these methods that enable structured specular surfaces to be measured successfully with high accuracy and speed in one setup. After a completed calibration, if the relative position of components (i.e., imaging sensors and screen) in the system remains unchanged, the subsequent measurement does not need to repeat the calibration process. These significantly improve the convenience and ability of measuring structured specular surfaces. In a SPMD configuration, only a fringe-displaying screen and two imaging sensors are required during a measurement process, which allows the configuration to be designed to be very compact. Note that this can reduce volume and weight of the measurement system. Therefore, significant advantages can be shown in application scenarios where embedded measurements are required (e.g., a diamond turning machine). Next step, the size of screen and imaging sensors will be further decreased and the configuration of SPMD will be optimized to achieve a more compact system for embedded measurement.

In addition to accuracy and speed, some improvements are required in SPMD. For example, currently the continuous segments are separated manually based on curvature difference, which leads to a low accuracy of segment edge selection and a difficulty to select segments with complex shapes. Moreover, manual segmentation severely limits the speed of measurement and calculation. However, we believe that as more algorithms (e.g., image recognition and matching algorithm, and artificial intelligence algorithm) are applied to assist the segmentation process in the future, SPMD will be more automated, and finally achieve the ambitious target of real-time measurement and display.

5 Conclusion

In this paper, we present a technique named segmentation phase measuring deflectometry which shows significant advantages in measurement of structured specular surfaces compared with traditional PMD techniques. We describe concept “segmentation,” spatial positioning technique and data fusion strategy in SPMD. By applying the developed techniques, the proposed technique can successfully measure 3D form of structured specular surface. At the same time, continuous form measurement accuracy of SPMD can keep as good as traditional GPMD. Experiments verified measurement ability of this technique to measure structured specular surfaces. Measurement accuracy, speed, and future development of SPMD, and its advantages compared with the existing measurement techniques in industry are discussed. Segmentation accuracy and speed are main factors affecting the SPMD performance. An automatic, fast, and accurate segmentation method will be explored in next step.