A framework for generalizable neural networks for robust estimation of eyelids and pupils

Abstract

Deep neural networks (DNNs) have enabled recent advances in the accuracy and robustness of video-oculography. However, to make robust predictions, most DNN models require extensive and diverse training data, which is costly to collect and label. In this work, we seek to improve the codevelop pylids, a pupil- and eyelid-estimation DNN model based on DeepLabCut. We show that performance of pylids-based pupil estimation can be related to the distance of test data from the distribution of training data. Based on this principle, we explore methods for efficient data selection for training our DNN. We show that guided sampling of new data points from the training data approaches state-of-the-art pupil and eyelid estimation with fewer training data points. We also demonstrate the benefit of using an efficient sampling method to select data augmentations for training DNNs. These sampling methods aim to minimize the time and effort required to label and train DNNs while promoting model generalization on new diverse datasets.

Appendices

Appendix A: Data augmentation

To simulate outdoor eye images, we used image processing techniques to artificially add perturbations to eye images in our training dataset by adding exposure, reflection, defocus blur, eye rotation, JPEG compression, motion blur, Gaussian noise, and adding mock pupils and glints to eye images. For each perturbation, we generated four additional images with increasing perturbation intensities. We detail the process of generating perturbed eye images below.

• 1. Exposure: To simulate the effect of an increase in exposure and decrease in the contrast between the pupil/eyelashes and other regions in the eye video in bright sunlight, we added four steps of luminance increments (each 35 units) to all pixels in each frame. After each increment, pixel values were clipped to a maximum of 255.

• 2. Rotation To simulate the effect of different camera angles and facial anatomy across participants, we rotated the eye videos in four five-degree increments followed by scaling and cropping to ensure uniform frame size. This rotation resulted in the eye going partially out of the frame for the 15 and 20-degree rotation conditions.

• 3. Reflection: Corneal reflection and shadows on the eye present a challenge while recording eye videos outdoors. We used the method presented in (Eivazi et al., 2019) to add reflections and shadows to the eye images. We modified the blending factor for images superimposed on the eye video in four steps. For every frame, we randomly selected the reflected image from the Driving Events Camera Dataset (Rebecq et al., 2019) which contains videos from dashboard cameras of cars driving through highways and cityscapes.

• 4. JPEG artifacts: Compressed video formats are desirable when storing eye videos as they take up less space. Thus, we tested the robustness of our DNN to compression artifacts by altering the video frames with JPEG compression. We varied the JPEG quality parameter (which varies from 100 to 0, denoting best to worst quality) from 32 to 8 in four steps of 8.

• 5. Defocus blur: To mimic the defocus blur from a camera we used the imgaug image augmentation library (Jung et al., 2020) and iteratively increased the severity parameter from 1 to 4 to create an incremental loss of focus in the eye videos.

• 6. Motion blur: To simulate the motion blur due to saccadic eye movements or blinks we used the motion blur as implemented in (Jung et al., 2020). Motion blur was varied between intensities 20 and 80. The angle for the direction for motion blur was randomly sampled from within the range [-45, 45] degrees.

• 7. Gaussian noise: To simulate the effect of Gaussian Noise we used the add Gaussian noise function as implemented in (Jung et al., 2020). We added Gaussian noise to eye images sampled once per pixel from a normal distribution ranging from N(0, 0.1*255) to N(0, 0.3*255).

• 8. Mock pupils: Often pupil like structures appear on eye images due to shadows, reflections based on the environment to avoid spurious detection of pupils due to such structure we augmented our data with blacked out mock pupils inspired bey (Eivazi et al., 2019). We created blacked out ellipses to simulate mock pupils, the center of the ellipse was sampled from a 2D Gaussian distribution centered on the center of the frame and spanning half the height and width of the eye image.

• 9. Mock glints: Using the same procedure as mock pupils, we created white ellipses to simulate the appearance of glints due to reflections and environmental conditions.

Appendix B: Multi-dimensional scaling

Fig. 9

Eyelid polynomial fits and ellipse fits for pupils demonstrating robustness of our baseline pylids model on samples from the GiW dataset. A Each row represents data from a different participant from the test data. Pupil positions as estimated by our pylids model are shown in red and compared to pupil estimates using the Pupil Labs package in blue. B and C compare pylids estimated pupil positions (red) to Pupil Labs package based pupil estimation (blue) during a blink (B) and a saccade (C). Partial pupil occlusion during a blink (B) leads to loss of data when using Pupil Labs pupil detection but not with pylids

Appendix C: Comparison with alternate model

Fig. 10

Correlation of cosine distance between GiW training data and test samples from the Fuhl dataset with average error and average keypoint likelihood for eyelid estimation. With an increase in Cosine distance there is monotonic increase in the average error

Appendix D: Distance from training distribution is correlated with model performance

First using our baseline pylids model trained on 520 labels from the Gaze in Wild dataset we evaluated if model performance is dependent upon cosine distance from the training dataset in ResNet50 space. To test this, we computed the rank order correlation between the model error on the test distribution and the Cosine distance from the training distribution in ResNet50 space (see Fig. 10).

Based on our finding that distance from the training distribution is correlated with model performance we used our guided sampling algorithm to select uniformly distributed samples to fine tune our model. We show that models fine-tuned using guided sampling outperform models trained using other sampling methods (see Fig. 6). To evaluate that the relationship between distance from the training distribution and model performance holds for the fine-tuned models we correlated the model error with distance from the training distribution (which now included new labeled frames.) We found that there is significant correlation and the relationship still holds true (see Fig. 11)

Fig. 11

Correlation between model error and cosine distance from training data for models fine-tuned using samples from the Fuhl dataset. The subplots are in the same order as 6. The correlation is significant across sampling methods

To further investigate if guided sampling did indeed result in reduction in distance from the training distribution we plotted the model error as a function of both distance to the initial baseline training samples and newly labeled samples. Compared to other sampling methods guided sampling reduced the distance from the training distribution for all test samples (see Fig. 12).

Fig. 12

Model error as a function of both the cosine distance from the initial training dataset and the new selected frames for training based on the Fuhl dataset. These subplots correspond to the same subplots visualized in Figs. 6 and 11. The dashed diagonal line represents the identity line. More samples below the diagonal shows that samples are closer to the new labeled training frames than the initial baseline training data. This is especially true for guided sampling