Quantitative Stereovision in a Scanning Electron Microscope

Abstract

Accurate, 3D full-field measurements at the micron-level are of interest in a wide range of applications, including both facilitation of mechanical experiments at reduced length scales and accurate profiling of specimen surfaces. Scanning electron microscope systems (SEMs) are a natural platform for acquiring high magnification images for stereo-reconstruction. In this work, an integrated methodology for accurate three-dimensional metric reconstruction and deformation measurements using single column SEM imaging systems is described. In these studies, the specimen stage is rotated in order to obtain stereo views of the specimen as it undergoes mechanical or thermal loading. Simulations and preliminary experimental studies at 300× demonstrate that (a) spatially-varying image distortions can be removed from images using a non-parametric distortion model, (b) the system can be reliably calibrated using distortion-corrected images of a planar object and grid at various orientations and (c) specimen rotation variability during the measurement phase can be controlled so that baseline strain errors are within the range of ±150 µε. Benchmark rigid body motion experiments using calibrated SEM views demonstrate that all components of strain in the reconstructed object have a mean value around O(10⁻⁴) and a random spatial distribution with standard deviation ≈ 300 micro-strain.

Appendices

Appendix A: Extrinsic Parameters Constraint

The calibration process described in “System Calibration and Distortion Removal” for the extrinsic parameters requires solving for both the extrinsic parameters and the 3D positions with known intrinsic parameters and image point correspondences (e.g., motion analysis in the computer vision literature). Theoretically, as described in previous work [27], when the intrinsic parameters are known it can be shown that there are 5 constraints on the six extrinsic parameters. The constraints can be obtained by using the essential matrix, [E], and the following formula [28]:

$$ \left[ {\mathbf{R}} \right] \cdot {\left[ {\mathbf{t}} \right]_\times } = \left[ {\mathbf{E}} \right] $$

(A.1)

where [R], t representing the rotation matrix and translation vector defined in eq. (2), respectively, with ${\mathbf{t}} = \left\{ \begin{gathered} {{\text{t}}_{\text{x}}} \hfill \\ {{\text{t}}_{\text{y}}} \hfill \\ {{\text{t}}_{\text{z}}} \hfill \\ \end{gathered} \right\}{; }\;\;{\left[ {\mathbf{t}} \right]_\times } = \left( {\begin{array}{*{20}{c}} 0 & { - {t_z}} & {{t_y}} \\ {{t_z}} & 0 & { - {t_x}} \\ { - {t_y}} & {{t_x}} & 0 \\ \end{array} } \right)$.

By construction, the essential matrix is a function of the unknown extrinsic parameters. Determination of [E] provides sufficient information to obtain the extrinsic parameters. The essential matrix can be determined using a series of equations. First, [E] can be used to relate the sensor positions of corresponding image points.

$$ {\left( {{{{\mathbf{{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}} {m}}}}}_{2i}}} \right)^T} \cdot \left[ {\mathbf{E}} \right] \cdot \left( {{{{\mathbf{{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}} {m}}}}}_{1i}}} \right) = 0 $$

(A.2)

where ${{\mathbf{{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}} {m}}}}_{1i}}$ and ${{\mathbf{{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}} {m}}}}_{2i}}$ in metric coordinates can be obtained from the image coordinate using the intrinsic parameters. To obtain the terms in the essential matrix, equation (A.2) can be employed with at least 5 point correspondences to obtain five independent equations. In addition to these equations, the three known constraints on the essential matrix can also be employed,

• i)
  $$ \det \left( {\left[ {\mathbf{E}} \right]} \right) = 0 $$

• ii) Employ the two Kruppa equations [27] with known intrinsic parameters.

In this form, [E] can be obtained up to an arbitrary non-dimensional scale, μ. The non-dimensional scale, μ, typically is associated with the vector defining the location of the pinhole in the second camera relative to the first camera so that t is the true translation vector obtained when the scale, μ, is uniquely determined.

Assuming that the scale is embedded in the translation vector, then the following formulae demonstrates that the scale also affects the 3D position of the corresponding point.

$$ {\hbox{M}} = {\left( {{{\mathbf{Q}}^{\mathbf{T}}}{\mathbf{Q}}} \right)^{{\mathbf{- 1}}}}{{\mathbf{Q}}^{\mathbf{T}}}{ }{\mathbf{b}} $$

(A.3)

where

$$ M = \left[ {\begin{array}{*{20}{c}} {{{\hbox{X}}_{\rm{W}}}} \\{{{\hbox{Y}}_W}} \\{{{\hbox{Z}}_W}} \\\end{array} } \right];{ }{\mathbf{b}} = \left[ {\begin{array}{*{20}{c}} {0} \\{0} \\{ - \left( {{{\hbox{t}}_{\rm{x1}}}{{\hbox{f}}_{\rm{x 2}}} + {{\hbox{t}}_{\rm{y1}}}{f_S}_2 + {{\hbox{t}}_{\rm{z1}}}\left( {{{\hbox{C}}_{\rm{x2}}} - {x_{\rm{S2}}}} \right)} \right)} \\{ - \left( {{{\hbox{t}}_{\rm{y1}}}{{\hbox{f}}_{\rm{y 2}}} + {{\hbox{t}}_{\rm{z1}}}\left( {{{\hbox{C}}_{\rm{y2}}} - {y_{\rm{S2}}}} \right)} \right)} \\\end{array} } \right] $$

$$ {\mathbf{Q}} = \left[ {\begin{array}{*{20}{c}} {{{\hbox{f}}_{\rm{x 1}}}} & {{{\hbox{f}}_S}_1} & {{{\hbox{C}}_{\rm{x1}}} - {x_{\rm{S1}}}} \\{0} & {{{\hbox{f}}_{\rm{y 1}}}} & {{{\hbox{C}}_{\rm{y1}}} - {y_{\rm{S1}}}} \\{{{\hbox{R}}_{11}}{{\hbox{f}}_{\rm{x 2}}} + {{\hbox{R}}_{{21}}}{f_{\rm{S2}}} + {{\hbox{R}}_{31}}({{\hbox{C}}_{\rm{x2}}} - {x_{\rm{S2}}})} & {{\hbox{R}}_{{12}}{{\hbox{f}}_{\rm{x 2}}} + {{\hbox{R}}_{{22}}}{f_{\rm{S2}}} + {\hbox{R}}_{{32}}({{\hbox{C}}_{\rm{x2}}} - {x_{\rm{S2}}})} & {{{\hbox{R}}_{13}}{{\hbox{f}}_{\rm{x 2}}} + {{\hbox{R}}_{{23}}}{f_{\rm{S2}}} + {{\hbox{R}}_{33}}\left( {{{\hbox{C}}_{\rm{x2}}} - {x_{\rm{S2}}}} \right)} \\{{{\hbox{R}}_{21}}{{\hbox{f}}_{\rm{y 2}}} + {{\hbox{R}}_{31}}\left( {{{\hbox{C}}_{\rm{y2}}} - {y_{\rm{S2}}}} \right)} & {{\hbox{R}}_{{22}}{{\hbox{f}}_{\rm{y 2}}} + {\hbox{R}}_{{32}}({{\hbox{C}}_{\rm{y2}}} - {y_{\rm{S2}}})} & {{{\hbox{R}}_{23}}{{\hbox{f}}_{\rm{y 2}}} + {{\hbox{R}}_{33}}\left( {{{\hbox{C}}_{\rm{y2}}} - {y_{\rm{S2}}}} \right)} \\\end{array} } \right] $$

Inspection of this form demonstrates that (a) the vector b is scaled by the parameter μ since each non-zero term contains a component of the vector t and (b) the matrix [Q] is not a function of the translation vector, but rather it is a function of the rotation tensor, the intrinsic camera parameters for both view 1 and view 2 and the sensor positions in views 1 and 2 for the common point. Taken together, eq. (A.3) demonstrates that each 3D position is also scaled by the parameter, μ, since each component is a linear function of the components of t.

Appendix B: Equations for Numerical Simulation of Strain Error Due to Pan-angle Variation

Without loss of generality, we consider the simplest case. Assuming a planar object, no distortion and zero skew, rotations are about x-axis and both translation vectors are [0 0 D]^T, we have the stereovision model [equation (2)] in a simplified form:

$$ \left\{ {\begin{array}{*{20}{c}} {{\rho_{1i}} \cdot {{\mathbf{m}}_{1i}} = \left[ {\mathbf{A}} \right] \cdot \left[ {{R_1}\;\;t} \right] \cdot {M_i}} \\{{\rho_{2i}} \cdot {{\mathbf{m}}_{2i}} = \left[ {\mathbf{A}} \right] \cdot \left[ {{R_2}\;\;t} \right] \cdot {M_i}} \\\end{array} } \right. $$

(B.1.a)

where ${\mathbf{m}} = \left[ {\begin{array}{*{20}{c}} {{x_s}} \\{{y_s}} \\1 \\\end{array} } \right],{\mathbf{M}} = \left[ {\begin{array}{*{20}{c}} {{X_w}} \\{{Y_w}} \\0 \\1 \\\end{array} } \right],\left[ {\mathbf{A}} \right] = \left[ {\begin{array}{*{20}{c}} {{f_x}} & 0 & {{C_x}} \\{} & {{f_y}} & {{C_y}} \\{} & {} & 1 \\\end{array} } \right],\left[ {\mathbf{R}} \right] = \left[ {\begin{array}{*{20}{c}} 1 & 0 & 0 \\0 & {\cos \theta } & { - \sin \theta } \\0 & {\sin \theta } & {\cos \theta } \\\end{array} } \right]$.with β = γ = 0 and the angle α for this case represented by θ.

Equation (B.1.a) can be written in the following,

$$ \left\{ {\begin{array}{*{20}{c}} {{{\mathbf{m}}_{1i}} = {\mathbf{f}}\left( {{f_x},{f_y},{C_x},{C_y},{\theta_1},D,{{\mathbf{M}}_i}} \right)} \\{{{\mathbf{m}}_{2i}} = {\mathbf{f}}\left( {{f_x},{f_y},{C_x},{C_y},{\theta_2},D,{{\mathbf{M}}_i}} \right)} \\\end{array} } \right. $$

(B.1.b)

For a series of points created in 3D space, M _i , the corresponding image position, m _1i and ${\tilde{m}_{2i}}$ are generated by

$$ \left\{ {\begin{array}{*{20}{c}} {{{\mathbf{m}}_{1i}} = {\mathbf{f}}\left( {{f_x},{f_y},{C_x},{C_y},{\theta_1},D,{{\mathbf{M}}_i}} \right)} \\{{{{\mathbf{\tilde{m}}}}_{2i}} = {\mathbf{f}}\left( {{f_x},{f_y},{C_x},{C_y},{{\tilde{\theta }}_2},D,{{\mathbf{M}}_i}} \right)} \\\end{array} } \right. $$

(B.2)

where ${\widetilde{\theta }_2}$ is the perturbed angle, ${\widetilde{\theta }_2} = {\theta_2} + \Delta \theta$. The biased 3D positions ${{\mathbf{\tilde{M}}}_i}$ (${{\mathbf{\tilde{M}}}_i} = {\left[ {\begin{array}{*{20}{c}} {{{\tilde{X}}_w}} & {{{\tilde{Y}}_w}} & {{{\tilde{Z}}_w}} & 1 \\\end{array} } \right]^T}$) due to the angular perturbation ∆θ are reconstructed from the images generated by eq. (B.2) using the least squares fitting procedure,

$$ {r_i} = \mathop {{\min }}\limits_{{{{\mathbf{\tilde{M}}}}_i}} \sum\limits_{j = 1}^2 {{{\left( {{{{\mathbf{\tilde{m}}}}_{ji}} - {\mathbf{f}}\left( {{f_x},{f_y},{C_x},{C_y},{\theta_j},D,{{{\mathbf{\tilde{M}}}}_i}} \right)} \right)}^2}} $$

(B.3)

where r _i is the residual error for the i-th point, ${{\mathbf{\tilde{m}}}_{1i}} = {{\mathbf{m}}_{1i}}$.

The artificial displacement field can then be computed by

$$ {\mathbf{dis}}{{\mathbf{p}}_i}\left( {\Delta \theta } \right) = {{\mathbf{\tilde{M}}}_i} - {{\mathbf{M}}_i} $$

(B.4)

and the strain field can be calculated as described previously [38].

Appendix C: Experimental Procedure for Stereovision in SEM

The experimental procedure includes (1) distortion correction, (2) system calibration, (3) specimen loading and (4) image processing to extract motion measurements.

Spatial Distortion Correction

A speckled planar target is placed in the SEM chamber on the eucentric goniometer, and oriented to be perpendicular to the e-beam. The target can be the experimental specimen if the pattern has good contrast in an SEM. Interruption of e-beam scanning is permitted during this stage, but all imaging parameters must be unchanged when initiating a new scan.

Once installed, the specimen is translated in both horizontal and vertical directions in the plane perpendicular to the optical axis (i.e. γ = 0°) following a cross-type path. Images are acquired at each translated position for use in distortion correction.

System Calibration

To calibrate an experimental setup, images are acquired of the micro-scale grid at various orientations. During the calibration process, the specimen is moved as necessary to maintain adequate focus. Table 7 presents the experimental out-of-plane rotations and in-plane rotations used to acquire 25 images of the micro-grid for calibration. Out-of-plane specimen rotations from −8° to 8° are performed using the eucentric goniometer. In-plane specimen rotations are performed using a built-in Quanta 200 electron beam rotation function.

Table 7 Image sequence for calibration

Experimental Phase

Once the distortion correction process and the calibration procedure have been completed, all SEM settings are maintained and the specimen is placed on the eucentric goniometer in the SEM chamber. Prior to applying load, images of the planar speckled specimen rotated out-of-plane by −10° and +10°, respectively, are acquired as the reference image pair.

Loading is applied to the specimen and images are acquired with the specimen oriented at −10° and +10° respectively. This process of out-of-plane specimen rotation and incremental loading is repeated until the experiment is completed.

Image Processing

After completing the three primary phases of the experimental work, images are evaluated from each phase.⁸ First, the spatial distortion correction images are processed using procedures outlined previously [32] for optical image correction. Second, all calibration images are analyzed to obtain the center-point position of each grid point. Each identified grid point is corrected for spatial distortion and bundle adjustment procedures within VIC-3D are used to obtain (a) the intrinsic parameters for the SEM stereo-vision system and (b) the orientation and position of each calibration grid.

Finally, all speckle images for the −10° and +10° specimen rotations are corrected for spatial distortion and then input into VIC-3D as stereo image pairs for cross-correlation and three-dimensional motion measurements.