Abstract
Background
A variety of imaging methods are available to obtain kinematic data at an interface, with a widely varying range of spatial and temporal resolution. These methods require a trade-off between imaging rate and resolution.
Objective
A deep learning framework trained on synchronous profilometry data acquired using two imaging modalities at two different spatial resolutions to enhance spatial resolution while maintaining temporal resolution is desired.
Methods
Fizeau interferometry (FIF) and frustrated total internal reflection (FTIR) are used to overcome the resolution-rate trade-off via a deep learning framework. The FTIR imaging data are recorded at high resolution, while the FIF imaging data are recorded with a lesser resolved, larger field of view. We apply a deep learning framework using a multi-layer convolutional neural network to enhance the FIF image resolution.
Results
With the deep learning framework, we achieve the high spatial resolution of measurements obtained by FTIR imaging in all three dimensions from the lower resolution FIF data. A high-order overset technique ultimately yields full up-scaled images from the network outputs without losing precision. The accuracy of the super-resolved image is evaluated using test data.
Conclusions
This hybrid framework, called HOTNNET, is implemented in its entirety on high-speed imaging profilometry data acquired in the study of droplet impacts on a smooth, solid surface, and is used to recover full, high-resolution images at high rates by unwrapping the phase of the interferometry. This framework can be readily adapted to other paired datasets by retraining the network on the novel data.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Myriad applications of imaging involve rapid phenomena [1,2,3,4]. In the studies of these phenomena, a trade-off between spatial and temporal resolution is imposed by the data bandwidth of typical high-speed camera hardware - we can obtain a pixel-dense image at low imaging rates, or low-pixel count images at high-rates, but we cannot obtain high pixel-count images at the highest frame rates without specialized techniques [5]. In addition to this technical constraint, physical phenomena such as optical diffraction or the finite numerical aperture of the image-forming objective also restrict spatial resolution. These limitations impede our ability to resolve rapid and multi-scale phenomena.
For rapid and multi-scale interfacial phenomena, interferometric imaging is often used as a means of visualization. This imaging modality has numerous applications in scientific and engineering fields, ranging from contact mechanics [6], fluid interfacial phenomena [7,8,9,10,11], biotechnology [12], nanotechnology [13], astronomy [14] to more industrial sectors like semiconductor [15] and display industries [16]. Interferometric imaging has the added benefit that it can be used as a non-invasive optical measurement technique. With interferometric imaging, the information about the target physical quantity is encoded in a two-dimensional fringe pattern, where the intensity of the fringes indicates the optical interference at a given spatial location within the image. This introduces a problem of lost phase information, leading to ambiguity in the interpretation of the interferogram; the ambiguity of the phase is only resolved in a process called demodulation. The demodulation process can limit the precision and accuracy of the measurement and requires a large field of view to work effectively.
Interferometry is often used in measurements of deformation during contact to obtain precise information about the gap between adjacent surfaces via interfacial profilometry; these measurements provide insight into material displacement, and can characterize material response due to mechanical loading or wear. As a result of the small spacing between adjacent surfaces, contact phenomena can occur on fleeting timescales; thus the problem of insufficient imaging resolution renders the aforementioned challenge of interferogram demodulation very difficult indeed. For fast interfacial dynamics, the limitations of high-speed imaging require that spatial resolution is reduced, or the field of view is shrunk, to obtain the necessary temporal resolution; however, any sacrifice of resolution or field of view can impair the demodulation effort by blurring or decreasing the fringe resolution. Even for slow or stationary interfacial profilometry, surface discontinuities, sharp interface changes and high slopes in the profilometry can degrade the fringe contrast [17] which makes it challenging to use interferometric optical techniques in nano-scale surface topographies [18].
Within the domain of contact mechanics, droplet impacts exemplify the rapid phenomena that can occur at an interface. Many experiments on droplet impacts use interferometry to directly visualize the contact profile and also the air gap between a liquid and a flat smooth substrate during impact [19,20,21]. Interferometric fringes can be demodulated to reveal the three dimensional profile of the air layer beneath the impacting drop, with a trough-to-peak resolution of approximately 150 nanometers between the dark and bright fringes. In these studies, however, absolute measurements of the air gap can only be obtained through more technically advanced implementations, such as white-light interferometry [8] or two-color techniques [22, 23], and require additional hardware and processing.
We address this imaging challenge using deep convolutional neural networks (DCNN) to obtain accurate and absolute values of the target quantities, while bypassing the need for explicit interferometric phase unwrapping. DCNNs were recently used to recover cohesive properties in the context of dynamic fracture mechanics from experimental interferometry images [24] in a manner similar to our recovery of FTIR imaging data from interferometric imaging data. Here, we exploit the proven ability of DCNNs [25] to obtain a super-resolved image from a low-resolution counterpart in the study of droplets impacting on a glass surface. Our DCNN, adapted from Visual Geometry Group (VGG) deep convolutional network [26], simultaneously demodulates the interferometric fringe patterns and maps them to target quantities. This builds upon recent efforts to use neural networks to unwrap the phase of interferometric fringes [27,28,29], that show better accuracy [30] and robustness [31] than approaches like phase shifting [32] and spatial phase demodulation methods [33].
A set of images recorded during the droplet impact experiments comprise the DCNN training database. We image the entire spatial extent of the droplet impact interface with an interferometric modality using a large field of view. Concurrently, we employ a frustrated total internal reflection (FTIR) microscopy modality to image a narrow sub-region of the interface at a higher resolution, producing the absolute target quantities. We then crop the interferometry images and the FTIR images to the common sub-region, thus forming the data pairs used to train and test the DCNN. Once trained, the network then maps all the other sub-regions to high resolution data; these upscaled subregions are then assembled via a high-order overset image reconstruction technique [34] for the entire sequence of low-resolution interferometric images. The hybrid processing framework, consisting of the High-order OverseT assembly method and the deep convolutional Neural NETwork (HOTNNET), produces demodulated, super-resolved images on a large field of view. We can then use a nanometer-precise, closed-form mapping function [35] to treat the HOTNNET output, thus obtaining accurate, absolute measurements with improved resolution along all directions, with accuracy of the order of the sub-fringe thickness resolution obtained with two-color interferometric imaging methods [36].
Methods
Imaging Setup
A 1.6 mm diameter drop of water-glycerol solution was released from a syringe luer adapter, and impacted on a smooth glass surface, as shown in Fig. 1(a) at top. The droplet viscosity in each experiment was selected from 8 different viscosities in the range of 1 to 100 centi-Stokes. Impact velocity was controlled by releasing the droplet from among 5 different fall heights, H, varying from 12.5 to 24.5 mm. At each height, all viscosities were tested once; thus, in total, 40 impact experiments were conducted. Prior to liquid-solid contact initiation, a nanometer scale air film forms between the droplet and the glass surface. The initial thickness of this air film varies inversely with impact velocity, from approximately 300 nm to tens of nm [37]. The liquid-solid-air interface was simultaneously imaged with two independent optical measurement techniques: FTIR and Fizeau Interferometry (FIF). These data were used in reports on the lift-off instability beneath an impacting droplet [38].
In the FTIR method [2, 37,38,39], the top surface of an optically smooth dove prism (BK-7 glass) is illuminated with collimated light, aligned to be incident at an angle greater than the critical angle for total internal reflection at a glass-air interface, but smaller than the critical angle at the glass-liquid interface. This light excites an exponentially decaying evanescent field above the prism’s surface, and when the liquid droplet enters within a few decay lengths of the surface, some light is transmitted to the droplet, and the intensity of the reflected light is less. The proportion of light transmitted to the drop increases as the droplet gets closer to the surface, further reducing the reflected intensity. The reflected light is imaged onto a high-speed camera sensor using a long-working distance (Mitutoyo 5X) microscope objective. The normalized light intensity is directly related to the air layer thickness by a deterministic transfer function, which varies with the incident light polarization, as described in Appendix 1.
The interferometric method requires a second optical path. In this approach, collimated light is directed onto the optical path using a 50-50 beam-splitter and focused onto the impact surface. When the droplet enters to within the coherence length of the light, an optical cavity is formed between the droplet surface and the solid surface. Some of the light reflects from the solid-air interface; a smaller amount of the transmitted light then reflects from the air-liquid interface. Depending on the gap thickness between the solid and the liquid, these two reflected beams will interfere [22, 40]. The interference between the two reflected beams from the spatially varying gap results in a pattern of fringes that are subsequently imaged onto a second high speed camera sensor. Both FTIR and FIF optical configurations are depicted in Fig. 1(a). Alignment of the optical paths and robust optical mounts were used to ensure image registration fidelity along the 40 experiments.
Simultaneous FIF and FTIR imaging is used in recent interfacial mechanics studies to probe the liquid-air interface beneath an impacting droplet [41,42,43]. These imaging modalities provide complimentary information about the liquid-air profile at different scales and with different resolutions - FTIR can resolve up to the wavelength of light with exponentially increasing resolution as the droplet approaches closer to the surface, whereas FIF can be used to measure gaps of up to several microns with lesser resolution. During simultaneous imaging, FTIR provides a reference for absolute measurement of the liquid-air interface with FIF; however, this doesn’t remove the need for FIF fringe demodulation.
In our simultaneous FIF and FTIR imaging setup, the FIF imaging path corresponds to a larger physical region extending over the entire impact zone beneath the droplet; by contrast, the FTIR modality images a smaller region near the impact axis with greater magnification. The common region recorded by both imaging methods is shown by a blue rectangular box in Fig. 1(b and c). The FIF images comprise \(128 \times 128\) pixels at a resolution of 12.7 microns per pixel in \(x_1\) and \(x_2\) directions with a precision of about one micron in the \(x_3\) direction. The FTIR images comprise \(512 \times 64\) pixels at a resolution of 3.8 by 5.0 microns per pixel in the \(x_1\) by \(x_2\) directions. FTIR resolves the gap thickness to within 10 nanometers in \(x_3\) direction [35]. FTIR and FIF images are recorded at 180,000 and 90,000 frames per second, respectively; example images are shown in Fig. 1(b and c). The sequence of images shown in Fig. 1(d and e) shows the rapid nature of contact formation.
Deep Convolutional Neural Network
We developed a deep convolutional neural network that maps our FIF images to our FTIR images. The network we employed [44] is adapted from the EnhanceNet neural network [26]. Similar networks have proven to be a robust tool in numerous applications in the field of data science whenever information is abundant [45, 46], and there is considerable interest in using these networks in engineering applications [24, 47, 48]. The structure of our network is detailed in Appendix 2 and is available online [44].
To train the network, we generate input-output data pairs from the images recorded during the impact experiments. Prior to training, the images are treated with a pre-processing protocol. First, the FIF images are interpolated in time using a quadratic interpolation function in order to synchronize the FTIR and FIF images. This yields a total of \(14 \, 871\) time synchronized frames from the 40 droplet impact experiments. Second, the data from the common region indicated by a blue rectangle in Fig. 1(b and c) are denoised and cropped from the images. The dimensions of the FIF and FTIR images are reduced to \(25 \times 108\) and \(64 \times 364\) pixels, respectively. Third, from each FIF-FTIR pair, 3 additional data pairs are generated by mirroring the images on \(x_1\), \(x_2\), and \(x_1 - x_2\) axes which increases the number of data pairs to \(59 \, 484\). As the final step, instead of pairing the FIF-FTIR images on a 1 to 1 basis, the input FIF data is augmented by 2 FIF frames, one before and one after the target time step, resulting in the pairing of 3 successive FIF images to 1 FTIR image; yielding data pairs with dimensions \(3 \times 25 \times 108 \Rightarrow 1 \times 64 \times 364\). The sequence of 3 FIF frames provides additional information to the network, facilitating the interferometric signal phase unwrapping and better capturing the dynamics of the liquid-air interface.
The total data pairs were split into 2 pools: the first pool consists of 32 impact experiments; images in this pool were used for both training/validation and testing purposes. The second pool consists of 8 separate droplet impact experiments; these images were used entirely for the final testing of the network performance. To train the network, 12,000 and 2000 data pairs were randomly drawn from the first pool to assemble the training and validation data sets, respectively. The testing data set for the network consists of the remaining of data pairs from the first pool, and all the pairs in the second pool. In total, 45,484 data pairs that the network did not see during the training were used to test the network’s performance. Out of these testing set, 12,395 pairs were from the second pool which were entirely left out during the network training. The root mean square error on all pixels averaged on 45,484 images is in order of \(10^{-6}\), as shown in Appendix 2.
There are four key stages of the droplet impact process, where the images appearing in each stage are somewhat distinctive; namely, before the contact initiation, immediately after the contact initiation, late stage of liquid-solid contact growth, and complete liquid-solid contact development. The last stage is discernible when the contact line(s) become stationary after sweeping the whole contact area and a single or multiple air bubbles might remain trapped. This stage of the impact process is distinguished by no perceptible change from frame-to-frame in the recorded images. Example network output along with the FIF and FTIR images for these four key stages is shown in Fig. 2. The network is able to capture the outline of the contact regions successfully at different stages of impact while simultaneously increasing the spatial resolution of the images. This “super-resolution” imaging is achieved by the network due to the higher magnification of the FTIR image. Additionally, the network enhances accuracy in the \(x_3\) direction as a natural consequence of the elevated resolution along \(x_3\) with the FTIR method. Thus, transforming the FIF data to the FTIR data by the use of deep learning implicitly bypasses the need for unwrapping the fringe patterns.
Mosaic Mapping with Overset Technique
The DCNN input dimension must match the size of the data that is used to train the network [49]. Our deep neural network is trained to transform subregions of the FIF images with dimensions of \(3 \times 25 \times 108\), converting them to high resolution \(64 \times 364\) FTIR-like images. However, we aim to recover the entire dynamics recorded with the full \(128 \times 128\) pixel field of view in the FIF image sequence. As a consequence of the mismatched dimensionality of our network input and the full FIF image, we must employ a means of using our trained network on the full FIF image despite this dimensional mismatch. Stitching the image sub-regions is commonly carried out in super-resolution imaging applications as a solution. In order to avoid losing the accuracy, we employed the overset grid technique [34, 50], which is inspired by Chimera method used extensively in high fidelity numerical simulations of fluid flows [51,52,53,54]. This method breaks a discretized domain into subregions, and provides a means of connecting the subregions by enforcing high order continuity conditions in an overlapping or overset neighborhood of each subregion’s boundary.
In order to implement the overset technique, we have partitioned the FIF data where there are a minimum of 2 and 3 overlapping pixels in the “untransformed” FIF images in \(x_1\) and \(x_2\) directions, respectively, which results in minimum of 5 overlapping data points in both \(\xi _1\) and \(\xi _2\) directions in the transformed subdomain as shown in Fig. 3(a). Using this partitioning we are thus breaking the entire FIF image into subregions; each of these subregions is an individual tile in the entire mosaic of the original FIF image. We can thus apply our neural network to each tile and obtain as an output a series of tiles that together form a new mosaic image representing the piece-wise mapping of the entire FIF image into a new super-resolved image. However, in order for the resulting super-resolved image to be coherent, the tiles must be connected properly. The procedure for connecting adjacent tiles in the super-resolved mosaic image is to interpolate the 2-dimensional equally-spaced coincident data points in the boundary region using a 6th-order Lagrangian interpolant [34]. The specific implementation of the interpolator stencils can be found elsewhere [44]. In this way, the subregions of the FIF data are fed one-by-one to the trained network, resulting in the high resolution mosaic shown in Fig. 3(b). Finally, by applying the pre-determined interpolator stencils the whole image is reconstructed, as shown in Fig. 3(c).
HOTNNET Results
Our DCNN can demodulate the interferometric fringe patterns and increase the data resolution of the image, as shown in Fig. 2. By merit of the FTIR image database used to train the network, the network also enhances the image resolution along the third axis, with some loss of depth-of-field. However, in the prior section we introduced the means to extend of the use of our DCNN to the entire FIF image using the high-order overset technique. The combined workflow of first using the DCNN region-by-region with the high-order overset technique comprises a hybrid processing framework that we call HOTNNET for short. With HOTNNET, we are able to enhance the resolution across the entire field of view contained in the interferometric images.
The results of HOTNNET are shown for 3 different FIF images corresponding to 3 stages of droplet impact event in Fig. 4. Upon the application of HOTNNET, instead of FIF fringe patterns, the reconstructed images contain light intensities representing the air gap measurements; such an output, in principle, provides a rich database for extraction of the physics of droplet impact, with images that can be mapped onto gap measurements. The HOTNNET output no longer suffer from the 150 nanometers ambiguity in the measurement that the monochromatic FIF and its accuracy is FTIR imaging modality which is proven to have an accuracy of a few nanometers in the \(x_3\) direction [35]. Using HOTNNET, we retain the resolution enhancement of the DCNN for each individual tile of the mosaic - the \(128 \times 128\) images are up-sampled to \(431 \times 328\) pixels.
Notably, any defect in the imaging system, e.g., dust on the camera sensor is removed by HOTNNET as indicated in Fig. 4(a–c). This is likely due to the use of 3 successive frames in the network training. Even in the presence of such artifacts, the output resolution of HOTNNET is at the same level as the FTIR imaging, suggesting that HOTNNET is capable of removing static noise from the input images.
HOTNNET successfully demodulates the grayscale and the profile edges of recorded FIF images. Some minor spurious noise is visible in Fig. 4(a and b), arising from the very rapid droplet spreading and the fastest dynamic contact line movement in these experiments. The recognition of fringe patterns is not as precise in Fig. 4(c), but the trapped air pockets (light spots in the dark area) are accurately identified. The lower quality of unwrapped fringes in this case arises from the lack of similar patterns in the network training data.
The overset technique employed in HOTNNET ensures that the reconstructed images do not show any non-uniformity in the inter-boundary regions of the tiles, thus eliminating the so-called ‘edge effect.’ as shown in Fig. 5. The high order stencils of HOTNNENT ensures that even at the edges the error is in the orders of \(10^{-4}\), but is larger very close to the liquid-solid contact due to sharp change of intensity.
Discussion
In this work, we introduce HOTNNET - a computational framework that upscales and demodulates interferometric images, thus enhancing spatial resolution in all three dimensions. At the core of HOTNNET is a deep convolutional neural network that eliminates static artifacts from the images while simultaneously enhancing resolution. HOTNNET uses a high order overset technique to produce full super-resolved images from the interferometric images of a given size and is not constrained by the dimensions of the initial network training sets. In order to demonstrate the HOTNNET framework presented in this paper, we used a total \(21 \, 888\) number of frames generated from experiments on droplet impact interfaces. The underlying physical results are explained elsewhere [35, 38] - when the drop approaches the surface, even at low speeds the initial dynamics of wetting are strongly altered by the confined air layer, which is displaced by the wetting fluid and deforms the liquid-air interface ahead of the contact line. The deformation results in a capillary disturbance that precedes the wetting front and leads to peculiar low wetting velocities beneath the impacting drop. These dynamics, ahead of the contact line, are inertial. Thus, the wetting front is surfing on a capillary wave and dragging a viscous tail in its wake.
Close to the edge of the contact line there is a distinguished halo region with sharp gradient in air layer thickness (Fig. 2(b and c)). In this vicinity, the FIF images suffer from degradation of fringe patterns due to lack of lateral resolution and sharp slopes at the liquid-air and liquid-solid interfaces. This introduces inaccuracies in capturing the actual peak/trough locations (iso-intensity contours) and their intensity magnitude. Consequently, fringe pattern demodulation becomes impossible or extremely inaccurate. In HOTNNET results, the outline of the halo section is captured and there is little ambiguity in the extent of the actual peak of halo region as the relation between the air layer thickness and the normalized light intensity is monotonic. However, the mapping function from FTIR to air layer thickness given in Appendix 1 may need correction [55] for high slopes and curvatures.
While the HOTNNET framework can be readily adapted to other imaging datasets, including those obtained with other interferometric imaging modalities for instance, the weight coefficient factors embedded in our trained neural network correspond to our specific optical setup. Consequently, changing the imaging system will require the network to be retrained. Retraining the network with a novel imaging system requires the collection of sufficient data for training, where the exact volume of training data depends on the quality of the images, but as few as several hundreds of images could suffice. The open-source code for HOTNNET including that for training the network is available on github [44].
Most neural network based super-resolution techniques use an artificially generated database of down-sampled, high-resolution images. Real world imaging artifacts such as sensor noise and static defects are absent from the images in these databases and thus these networks cannot be used for quantitative imaging modalities in science and engineering. HOTNNET faithfully treats the real-world recorded images because the complete data pairs for its network training are the outcome of laboratory experiments including imaging artifacts.
HOTNNET generates the final super-resolved mosaic by first mapping local tiles that are each super-resolved with the deep convolutional neural network. Because the regions of the mosaic correspond to different stages of the dynamics, the local mapping is non-invertible; thus on some level the network is classifying the contents of each tile based on limited information. Thus, like all other supervised learning strategies, the neural network in HOTNNET may lose fidelity when it is subject to substantially different interferometric image inputs. Nevertheless, for sufficiently similar interferometric images, HOTNNET is capable of producing super-resolved data that might be used quantitatively, and represents a step toward physically relevant data generated by neural network. Indeed, given the universality of physical behaviors, it is not outside the realm of possibility to use data obtained with HOTNNET to evaluate physical principles; this would require further work and verification to ensure that results obtained using the HOTNNET output are not artifacts of the processing, but instead are physical phenomena.
Conclusion
HOTNNET could be directly applied in fields of precision metrology provided that the neural network is appropriately trained. Using HOTNNET, we address constraints on our measurement due to the performance characteristics of our imaging hardware; however one can envision a means by which HOTNNET could address other constraints. For example, if an imaging system places an elliptical aperture in the imaging plane, the imaging domain would result in a non-rectangular tile shape. The overset technique in HOTNNET can be readily adapted to curvilinear geometries. Additionally, with appropriate calibration, HOTNNET could increase the accuracy of white light interferometry [56], which is currently limited by the bandwidth of the color filters on the CCD camera.
HOTNNET is designed to address constraints imposed by a given experimental setup using machine learning. This framework makes progress toward accurate and reliable physical measurements as the output of a trained neural network. Here, images recorded during droplet impact and contact formation were treated with HOTNNET, but the technique introduced here can be employed in the broader context of physics and engineering. HOTNNET thus offers a compelling and powerful tool that simultaneously achieves noise rejection, phase unwrapping and resolution enhancement of images when direct measurements are not possible or are cost prohibitive.
References
Zewail AH (2000) Femtochemistry: atomic-scale dynamics of the chemical bond. J Phys Chem A 104(24):5660–5694
Rubinstein SM, Cohen G, Fineberg J (2004) Detachment fronts and the onset of dynamic friction. Nature 430(7003):1005–1009
Versluis M (2013) High-speed imaging in fluids. Exp Fluids 54(2):1458
Crua C, Shoba T, Heikal MR, Gold MR, Higham C (2010) High-speed microscopic imaging of the initial stage of diesel spray formation and primary breakup. SAE 2010 Powertrains Fuels & Lubricants Meeting
Dillavou S, Rubinstein SM, Kolinski JM (2019) The virtual frame technique: ultrafast imaging with any camera. Opt Express 27(6):8112–8120
Johnson KL (1985) Contact mechanics. Cambridge University Press
Zdravkov AN, Peters GWM, Meijer HEH (2006) Film drainage and interfacial instabilities in polymeric systems with diffuse interfaces. J Colloid Interface Sci 296(1):86–94
van der Veen RCA, Tran T, Lohse D, Sun C (2012) Direct measurements of air layer profiles under impacting droplets using high-speed color interferometry. Phys Rev E 85:026315
de Ruijter MJ, Charlot M, Voué M, De Coninck J (2000) Experimental evidence of several time scales in drop spreading. Langmuir 16(5):2363–2368
de Ruiter J, van den Ende D, Mugele F (2015) Air cushioning in droplet impact. ii. experimental characterization of the air film evolution. Phys Fluids 27(1):012105
Dhaouadi W, Kolinski JM (2019) Bretherton’s buoyant bubble. Phys Rev Fluids 4(12):123601
Derin S, Oğuzhan A, Selim Ünlü M (2017) Quantitative interferometric reflectance imaging for the detection and measurement of biological nanoparticles. Biomed Opt Express 8(6):2976–2989
Kim I (Isaac), Paik S, Jeon YB, Park JS, Kim H, Kim H (2018) Full-field thickness measurement of ultrathin liquid film in receding contact-induced nano-channel using surface plasmon resonance. Opt Express 26(16):20975–20989
Monnier JD (2003) Optical interferometry in astronomy. Rep Prog Phys 66(5):789–857
Tignon J, Marquezini MV, Hasche T, Chemla DS (1999) Spectral interferometry of semiconductor nanostructures. IEEE J Quantum Electron 35(4):510–522
Lee T, Jeong H, Kim H, Lee S, Kim D (2016) Effect of platen shape on evolution of total thickness variation in single-sided lapping of sapphire wafer. Int J Precis Eng Manuf 3(3):225–229
Wu D, Fang F (2021) Development of surface reconstruction algorithms for optical interferometric measurement. Front Mech Eng 16(1):1–31
Vithin AVS, Ajithaprasad S, Rajshekhar G (2019) Step phase reconstruction using an anisotropic total variation regularization method in a diffraction phase microscopy. Appl Opt 58(26):7189–7194
Driscoll MM, Nagel SR (2011) Ultrafast interference imaging of air in splashing dynamics. Phys Rev Lett 107:154502
Liu Y, Tan P, Xu L (2013) Compressible air entrapment in high-speed drop impacts on solid surfaces. J Fluid Mech 716:R9
Li EQ, Thoroddsen ST (2015) Time-resolved imaging of a compressible air disc under a drop impacting on a solid surface. J Fluid Mech 780:636–648
de Ruiter J, Oh JM, van den Ende D, Mugele F (2012) Dynamics of collapse of air films in drop impact. Phys Rev Lett 108:074505
de Ruiter J, Mugele F, van den Ende D (2015) Air cushioning in droplet impact. I. Dynamics of thin films studied by dual wavelength reflection interference microscopy. Phys Fluids 27(1):012104
Jin H, Jiao T, Clifton RJ, Kim K-S (2022) Dynamic fracture of a bicontinuously nanostructured copolymer: a deep-learning analysis of big-data-generating experiment. J Mech Phys Solids 164:104898
Qiao C, Li D, Guo Y, Liu C, Jiang T, Dai Q, Li D (2021) Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat Methods 18(2):194–202
Sajjadi MSM, Scholkopf B, Hirsch M (2017) Enhancenet: single image super-resolution through automated texture synthesis. In: Proceedings of the IEEE International Conference on Computer Vision. pp 4491–4500
Zhong C, Gao Z, Wang X, Gao C, Yang S, Sun X, Wen X, Feng Z, Wang S (2018) The machine learning method of phase extraction in interferometry. Opt Lasers Eng 110:384–391
Li Z, Li X, Liang R (2020) Random two-frame interferometry based on deep learning. Opt. Express 28(17):24747–24760
Black J, Chen S, Thomas JG, Zhu Y (2020) Deep neural networks for efficient phase demodulation in wavelength shifting interferometry. Optics Communications 475:126250
Feng S, Chen Q, Gu G, Tao T, Zhang L, Yan H, Yin W, Zuo C (2019) Fringe pattern analysis using deep learning. Adv Photonics 1(02):1
Lee S, Oh O, Kim Y, Kim D, Hussey DS, Wang G, Lee SW (2020) Deep learning for high-resolution and high-sensitivity interferometric phase contrast imaging. Sci Rep 10(1):9891
Zuo C, Feng S, Huang L, Tao T, Yin W, Chen Q (2018) Phase shifting algorithms for fringe projection profilometry: a review. Opt Lasers Eng 109:23–59
Su X, Zhang Q (2010) Dynamic 3-D shape measurement method: a review. Opt Lasers Eng 48(2):191–204
Sherer SE, Scott JN (2005) High-order compact finite-difference methods on general overset grids. J Comput Phys 210(2):459–496
Kolinski JM, Kaviani R, Hade D, Rubinstein SM (2019) Surfing the capillary wave: wetting dynamics beneath an impacting drop. Phys Rev Fluids 4:123605
de Ruiter J, Lagraauw R, van den Ende D, Mugele F (2014) Wettability-independent bouncing on flat surfaces mediated by thin air films. Nat Phys 11:48
Kolinski JM, Mahadevan L, Rubinstein SM (2014) Drops can bounce from perfectly hydrophilic surfaces. Europhys Lett 108(2):24001
Kolinski JM, Mahadevan L, Rubinstein SM (2014) Lift-off instability during the impact of a drop on a solid surface. Phys Rev Lett 112(13):134501
Kolinski JM, Rubinstein SM, Mandre S, Brenner MP, Weitz DA, Mahadevan L (2012) Skating on a film of air: drops impacting on a surface. Phys Rev Lett 108(7):074503
Driscoll MM, Stevens CS, Nagel SR (2010) Thin film formation during splashing of viscous liquids. Phys Rev E 82(3):036302
Pack M, Hu H, Kim D, Zheng Z, Stone HA, Sun Y (2017) Failure mechanisms of air entrainment in drop impact on lubricated surfaces. Soft Matter 13:2402–2409
Pack M, Kaneelil P, Kim H, Sun Y (2018) Contact line instability caused by air rim formation under nonsplashing droplets. Langmuir 34(17):4962–4969
Chantelot P, Lohse D (2021) Drop impact on superheated surfaces: short-time dynamics and transition to contact. J Fluid Mech 928:A36
Kaviani R, Kolinski JM (2019) HOTNNET. https://gitlab.epfl.ch/kaviani/hotnnet
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436
Deng L, Yu D (2014) Deep learning: methods and applications. Found Trends Signal Process Copy 7(3–4):197–387
Dimiduk DM, Holm EA, Niezgoda SR (2018) Perspectives on the impact of machine learning, deep learning, and artificial intelligence on materials, processes, and structures engineering. Integr Mater Manuf Innov 7(3):157–172
Hoffmann J, Bar-Sinai Y, Lee LM, Andrejevic J, Mishra S, Rubinstein SM, Rycroft CH (2019) Machine learning in a data-limited regime: augmenting experiments with synthetic data uncovers order in crumpled sheets. Sci Adv 5(4)
Yue D, Wang C, Zhang C, Guo L, Chen Y, Yan M, Feng Q, Shang M, Kuang W, Wang Z, Huang Z-L (2021) Computational framework for generating large panoramic super-resolution images from localization microscopy. Biomed Opt Express 12(8):4759–4778
Chesshire G, Henshaw WD (1990) Composite overlapping meshes for the solution of partial differential equations. J Comput Phys 90(1):1–64
Kaviani R, Afshari A, Nikkhah-Bahrami M, Ebrahimi M (2011) LES of a nozzle/jet aeroacoustics. International Congress on Sound and Vibration, ICSV 1:333–340
Wang G, Duchaine F, Papadogiannis D, Duran I, Moreau S, Gicquel LYM (2014) An overset grid method for large eddy simulation of turbomachinery stages. J Comput Phys 274:333–355
Bodony DJ, Zagaris G, Reichert A, Zhang Q (2010) Aeroacoustic predictions in complex geometries. Procedia Engineering 6:234–243
Bodony DJ, Zagaris G, Reichert A, Zhang Q (2011) Provably stable overset grid methods for computational aeroacoustics. J Sound Vib 330(17):4161–4179
Hentschel M, Schomerus H (2002) Fresnel laws at curved dielectric interfaces of microresonators. Phys Rev E 65:045603
Kumar UP, Haifeng NW, Mohan K, Kothiyal MP (2012) White light interferometry for surface profiling with a colour CCD. Opt Lasers Eng 50(8):1084–1088
Shirota M, van Limbeek MA. J, Lohse D, Sun C (2017) Measuring thin films using quantitative frustrated total internal reflection (FTIR). Eur Phys J E 40(5):54
Zhu S, Yu AW, Hawley D, Roy R (1986) Frustrated total internal reflection: a demonstration and review. Am J Phys 54(7):601–607
Chubynsky MV, Belousov KI, Lockerby DA, Sprittles JE (2020) Bouncing off the walls: the influence of gas-kinetic and van der Waals effects in drop impact. Phys Rev Lett 124:084501
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Preprint at https://arxiv.org/abs/1409.1556
Acknowledgements
The authors are grateful for helpful comments from Shmuel M. Rubinstein and Dr. Navid Borhani.
Funding
Open access funding provided by EPFL Lausanne.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of Interest
The authors declare no conflicts of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1
Profilometry by FTIR Imaging Modality
Profilometry with FTIR image processing is described in [57]. To summarise, firstly the FTIR image is stretched in the \(x_2\) direction by a factor of
to compensate for the optical transformation in the total internal reflection; where \(n_G\), \(n_S\), and \(\phi _1\) are refractive index of air (gas), BK-7 glass (substrate) and the light angle of incident at the air-substrate interface, respectively. Second, the image intensities are divided by the background image to obtain normalized intensities,
Finally, the normalized intensities are mapped into absolute air layer thickness fields, \(h(x_1,x_2)\), using the following equations.
where \(\delta\) is resulted based on Fresnel equations from the following relation [58]
with constant values
In these equations, \(n_L\) is the refractive index of the liquid, j is the imaginary unit, and the subscripts \(\perp /\parallel\) denote s- and p-polarized components of the light, respectively. In practice, either pure s- or p- polarized light is used in the FTIR imaging to simplify the equations above and h is readily deduced using using a lookup table [57], instead of inverting relation (4).
In our droplet impact experiments, we are specifically interested in air layer thicknesses of nanometer scales where the van der Waals and other short-range interfacial forces can drive contact formation; these are the subject of recent numerical investigations [59]. We have targeted an accuracy close to few nanometers of precision. Close to darkest point of the camera (\(I=O(10)\)) the slope of the transfer function gets over \(10^3\) nanometer per unit change of normalized intensity (\(\hat{I}\)). Hence, in our overall HOTNNET imaging profilometry, we kept normalized root mean-squared error (NRMSE) per pixel below \(10^{-5}\). However, our initial implementation of FTIR profilometry of droplet impact experiments [38] employed a simplified total energy inversion algorithm,
that assumed all light incident upon the droplet’s surface above the air transmitted into the drop, without taking into account the important role played by polarization of the light, resulting in error bars in Fig. 6 when doing the profilometry of the halo section. Figure 6 shows the kinematics of wetting front for the droplet with a viscosity of 76 centi-Stokes falling from a height of 18 mm.
Appendix 2
The Architecture of the the Neural Network
We developed a deep convolutional neural network that maps our FIF images to our FTIR images. The network we employed [44] is adapted from the EnhanceNet neural network [26], consisting of several interconnected layers, as depicted schematically in Fig. 7.
The EnhanceNet is a generative feed-forward fully convolutional network [26], derived from Visual Geometry Group (VGG) deep neural network [60], using strided convolutions but with leaky ReLU activation and without pooling layers. Our network uses an Adam optimizer with a learning rate of 0.0005 to minimize the mean square error cost function. Our goal was to achieve normalized root mean-squared error of order \(O(10^{-6})\) in the interior of the mosaics (see Appendix 1) which is 2 orders of magnitude smaller than the error on edges, i.e., \(O(10^{-4})\) (refer to Fig. 5).
With this metric, we found the EnhanceNet perform better in comparison with similar upsampling and convolution networks in terms of capturing the contact line (Fig. 8). Different networks were implemented using the TensorFlow version 1.14 Python library and was run on an NVIDIA Tesla T4 graphics processing unit. Our DCNN has shown better accuracy and an increased rate of convergence compared to GAN and U-Net. It took our modified EnhanceNet training 800 epochs over a course of 9.5 hours runtime to satisfy the convergence criterion. This was faster and more efficient compared to the original EnhanceNet and the VGG-16 architectures taking over 2400 and 4000 epochs with runtime period of each over 24 hours, respectively. Due to the fine structure of the liquid-solid contact line, as many training epochs were required for the network to achieve reproducibility of these dynamics. Our attempts for U-Net architecture training have been unsuccessful (Fig. 8(c)). We used batch shuffling of the training data set with a drop-out ratio of 12 percent was used to avoid over-fitting.
Further, here we investigate the image dimensions in convergence of training the network in HOTNNET. In an ideal scenario, a super-resolution microscopic imaging method should be capable of recording data with high-precision, covering a large spatial domain. In our experiment, the FIF imaging covers a large enough surface; using the HOTNNET technique, we have been able to improve the accuracy and quality of our FIF images to the same level as our locally recorded FTIR images as shown Fig. 9(b). We are further confident about the overset technique to remove the edge discontinuity (Fig. 5). For the physics of contact lines forming beneath impacting droplets, our imaging precision in \(x_3\) direction is satisfactory. However, further improvements of the accuracy in the \(x_1\) and \(x_2\) directions, from the current 3.9 and 5.5 microns per pixel, respectively, are desirable. This augmented resolution can be achieved by replacing the current 5X microscope objective in the FTIR setup to a greater magnification. This will decrease the actual dimensions of the subset used for the network training, i.e., the blue rectangle in Fig. 1 or the white rectangular region in Fig. 5. One must be aware that the smaller subset recorded with the greater magnification might not have sufficient information to train the network to demodulate the FIF fringe patterns. In the other words, the training subset must correspond to sufficient interferometry fringe variation for trainability of the network. This would likely require at least one full modulation from minimum to maximum intensity in the fringe pattern. If this criterion is satisfied, the network is trainable, the rest of the HOTNNET procedure is straightforward.
To quantitatively evaluate the minimum size of the training data, we have shrunk the dimensions of DCNN input images to \(108 \times 25\), \(97 \times 24\), \(86 \times 20\), \(76 \times 18\), \(65 \times 15\), \(54 \times 13\), and \(43 \times 10\) towards the center and checked the performance of the network on \(45 \, 484\) test data pairs which were unseen data sets during the network training in terms of the root mean square of error on all pixels. The architecture and the other parameters of the network, including the maximum number of epochs, were maintained to ensure direct comparability of the network fidelity. The results show that using larger dimensions reduces the error, reproducing the FTIR images much more satisfactorily. The network is not trainable for \(43 \times 10\) pixels, i.e., 0.57 mm \(\times\) 0.14 mm subdomain size. For \(54 \times 13\) pixels input, i.e., 0.71 mm \(\times\) 0.18 mm subdomain, which is approximately the subdomain size which can be covered by a 20X objective, the network is trainable but the error is found to be high. We use a maximum root mean square of \(10^{-4}\) as a threshold for trainability, which was achieved with a subdomain dimensions of \(86 \times 20 \approx 1700\) pixels, or 1.15 mm \(\times\) 0.28 mm tile size. This width of the tile is comparable to the diameter of the droplet, i.e., 1.6 mm.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Kaviani, R., Kolinski, J. High Resolution Interferometric Imaging of Liquid-Solid Interfaces with HOTNNET. Exp Mech 63, 309–321 (2023). https://doi.org/10.1007/s11340-022-00912-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11340-022-00912-z