Automated Bystander Detection and Anonymization in Mobile Photography

Abstract

As smartphones have become more popular in recent years, integrated cameras have seen a rise in use. This trend has negative implications for the privacy of the individual in public places. Those who are captured inadvertently in others’ pictures often have no knowledge of being included in a photograph nor have any control over how the photos of them might be distributed. To address this growing issue, we propose a novel system for protecting the privacy of bystanders captured in public photos. A fully automated approach to accurately distinguish the intended subjects of photos from strangers is first explored. To accurately distinguish these subjects and bystanders, we develop a feature-based classification approach utilizing entire photos. Additionally, we consider the privacy-minded case of only utilizing local face images with no contextual information from the original image by developing a convolutional neural network-based classifier. Considering the face to be the most sensitive and identifiable portion of a bystander, both classifiers are utilized to form an estimation of facial feature locations which can then be obfuscated to protect bystander privacy. We implement and compare three methods of facial anonymization: black boxing, Gaussian blurring, and pose-tolerant face swapping. To validate and explore the viability of these anonymization methods, a comprehensive user survey is conducted to understand the difference in appeal and viability between them.

Ang Li was a PhD student at the University of Arkansas at the time of this work.

Appendices

Appendix

A Performance Characteristics of Classifiers

Table 3. Average single prediction forward-pass runtime (Intel i9-10900k)

Table 3 shows the measured single forward-pass runtimes for each of the examined classifiers averaged over 1000 runs. All feature-based classifiers have almost negligible runtime requirements for prediction operations on an Intel i9 platform. This indicates they would be excellent candidates to run directly on resource-constrained mobile devices.

Fig. 8.

Progressive accuracy over training mini-batches for the CNN classifier.

Figure 8 shows the accuracy of the CNN as mini-batches progress during training. Taken in conjunction with Fig. 6, the CNN appears to only converge after the 220th mini-batch where loss and accuracy variance is lowest.

B Detailed Survey Results

Figure 9 shows detailed results of participant responses to survey questions 1–7. It is clear from these results that almost all respondents believe that photo privacy is important. Additionally, the rapid transition in willingness to use a system that does not degrade photo quality shows how paramount developing minimally invasive obfuscation methods is for real-world use.

Fig. 9.

Survey responses to opinion questions 1–7.

Fig. 10.

Survey responses to rating the impact of anonymization methods on photos.

Figure 10 shows that respondents generally believe that the blurring and swapping methods are the least impactful to the sample photos. Interestingly, there was not a strong correlation between responses here and with the next set of questions over user willingness to use these same methods on their own photos.

Fig. 11.

Survey responses to rating how willing users would be to use each anonymization method.

The responses on willingness to use the obfuscation methods (see Fig. 11) demonstrate that there are more factors in determining what users want to use than perceived impact to photo quality. For example, face swapping, while rated as generally having a low impact to photo quality, received many negative responses from users in their likelihood to actually use it.