Abstract
Discrimination in decision-making systems is of growing concern as machine learning techniques (especially deep learning) are increasingly applied in systems with societal impact. Multiple recent works have proposed to identify/generate discriminative samples through fairness testing. State-of-the-art fairness testing methods can efficiently generate many discriminative samples, which can be subsequently used to improve the fairness of the model. Unfortunately, the applicability of these approaches is limited in practice as they require the availability of both the model and the training data, i.e., a white-box setting. In a black-box setting (e.g., testing online services), existing approaches are impractical for multiple reasons, e.g., they require huge testing budgets. In this work, we propose a black-box fairness testing approach for neural networks, namely BREAM, which addresses two challenges, i.e., how to generate many discriminative samples without querying many times and how to guide the searching without the original model. Our overall idea is to obtain approximate gradients by training shadow models to effectively guide the discriminative sample generation for black-box DNNs. We also observe the density diversity of the distribution of discrimination, which enables incremental maintenance of shadow models and rational allocation of search resources by dividing multiple subspaces. We evaluated BREAM on three widely adopted datasets for fairness research. The results show that BREAM achieves a 9X higher performance than existing black-box methods, comparable to the state-of-the-art white-box fairness method.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Code of black-box fairness testing with shadow models. https://github.com/lenijwp/Black-box-Discrimination-Finder
Embedding api-build faster great ai algorithms for hr. https://hrflow.ai/embedding/
Aggarwal, A., Lohia, P., Nagar, S., Dey, K., Saha, D.: Black box fairness testing of machine learning models. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 625–635 (2019)
Angwin, J., Larson, J., Mattu, S., Kirchner, L.: Machine bias. ProPublica, 23 May 2016
Batina, L., Bhasin, S., Jap, D., Picek, S.: CSI NN: reverse engineering of neural network architectures through electromagnetic side channel. In: 28th USENIX Security Symposium (USENIX Security 19), pp. 515–532. USENIX Association, Santa Clara, CA, August 2019. https://www.usenix.org/conference/usenixsecurity19/presentation/batina
Bojarski, M., et al.: End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 (2016)
Cherian, V., Bindu, M.: Heart disease prediction using Naive Bayes algorithm and laplace smoothing technique. Int. J. Comput. Sci. Trends Technol. 5(2), 68–73 (2017)
Chollet, F., et al.: Keras (2015). https://github.com/keras-team/keras
Corbett-Davies, S., Goel, S.: The measure and mismeasure of fairness: a critical review of fair machine learning. arXiv preprint arXiv:1808.00023 (2018)
Dua, D., Graff, C.: UCI adult data set (2017). https://archive.ics.uci.edu/ml/datasets/adult
Dua, D., Graff, C.: UCI bank marketing data set (2017). https://archive.ics.uci.edu/ml/datasets/bank+marketing
Dua, D., Graff, C.: UCI machine learning repository (2017). https://archive.ics.uci.edu/ml
Dua, D., Graff, C.: UCI statlog (german credit data) data set (2017). https://archive.ics.uci.edu/ml/datasets/statlog+(german+credit+data)
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 214–226 (2012)
Fan, M., Wei, W., Jin, W., Yang, Z., Liu, T.: Explanation-guided fairness testing through genetic algorithm. In: Proceedings of the 44th International Conference on Software Engineering, pp. 871–882 (2022)
Feldman, M., Friedler, S.A., Moeller, J., Scheidegger, C., Venkatasubramanian, S.: Certifying and removing disparate impact. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 259–268 (2015)
Ferral, K.: Wisconsin supreme court allows state to continue using computer program to assist in sentencing. the capital times, 13 July 2016
Galhotra, S., Brun, Y., Meliou, A.: Fairness testing: testing software for discrimination. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pp. 498–510 (2017)
Goh, G., Cotter, A., Gupta, M., Friedlander, M.P.: Satisfying real-world goals with dataset constraints. In: Advances in Neural Information Processing Systems, pp. 2415–2423 (2016)
HLEG, A.: High-level expert group on artificial intelligence. Ethics Guidelines for Trustworthy AI (2019)
Huang, G., Li, Y., Wang, Q., Ren, J., Cheng, Y., Zhao, X.: Automatic classification method for software vulnerability based on deep neural network. IEEE Access 7, 28291–28298 (2019)
Kamiran, F., Calders, T., Pechenizkiy, M.: Discrimination aware decision tree learning. In: 2010 IEEE International Conference on Data Mining, pp. 869–874. IEEE (2010)
Kamishima, T., Akaho, S., Asoh, H., Sakuma, J.: Fairness-aware classifier with prejudice remover regularizer. In: Flach, P.A., De Bie, T., Cristianini, N. (eds.) ECML PKDD 2012. LNCS (LNAI), vol. 7524, pp. 35–50. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33486-3_3
Liang, L., Zheng, K., Sheng, Q., Huang, X.: A denial of service attack method for an IoT system. In: 2016 8th International Conference on Information Technology in Medicine and Education (ITME), pp. 360–364. IEEE (2016)
Lloyd, S.P.: Least squares quantization in PCM. IEEE Trans. 28(2), 129–137 (1982)
Oh, S.J., Augustin, M., Schiele, B., Fritz, M.: Towards reverse-engineering black-box neural networks (2018)
Olson, P.: The algorithm that beats your bank manager. CNN money, 15 March 2011
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017)
Ribeiro, M.T., Singh, S., Guestrin, C.: Why should i trust you? Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)
Ribeiro, M.T., Wu, T., Guestrin, C., Singh, S.: Beyond accuracy: behavioral testing of NLP models with checklist. arXiv preprint arXiv:2005.04118 (2020)
Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)
Schuba, C.L., Krsul, I.V., Kuhn, M.G., Spafford, E.H., Sundaram, A., Zamboni, D.: Analysis of a denial of service attack on TCP. In: Proceedings. 1997 IEEE Symposium on Security and Privacy (Cat. No. 97CB36097), pp. 208–223. IEEE (1997)
Sweeney, L.: Discrimination in online ad delivery. Queue 11(3), 10–29 (2013)
Szegedy, C., et al.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)
Tramer, F., et al.: Fairtest: discovering unwarranted associations in data-driven applications. In: 2017 IEEE European Symposium on Security and Privacy (EuroS &P), pp. 401–416. IEEE (2017)
Tramèr, F., Zhang, F., Juels, A., Reiter, M.K., Ristenpart, T.: Stealing machine learning models via prediction APIs. In: 25th \(\{\)USENIX\(\}\) Security Symposium (\(\{\)USENIX\(\}\) Security 16), pp. 601–618 (2016)
Udeshi, S., Arora, P., Chattopadhyay, S.: Automated directed fairness testing. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pp. 98–108 (2018)
Wang, B., Gong, N.Z.: Stealing hyperparameters in machine learning. In: 2018 IEEE Symposium on Security and Privacy (SP), pp. 36–52. IEEE (2018)
Yu, H., et al.: Cloudleak: large-scale deep learning models stealing through adversarial examples. In: Proceedings of Network and Distributed Systems Security Symposium (NDSS) (2020)
Zhang, B.H., Lemoine, B., Mitchell, M.: Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335–340 (2018)
Zhang, L., Zhang, Y., Zhang, M.: Efficient white-box fairness testing through gradient search. In: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 103–114 (2021)
Zhang, L., Wu, Y., Wu, X.: Achieving non-discrimination in data release. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1335–1344 (2017)
Zhang, P., et al.: White-box fairness testing through adversarial sampling. In: 42nd International Conference on Software Engineering (2020)
Zhang, P., et al.: White-box fairness testing through adversarial sampling (2020). https://github.com/pxzhang94/ADF
Acknowledgement
This research is supported by National Key Research and Development Program of China (2020YFB1406900), National Natural Science Foundation of China (U21B2018, 62161160337, U20B2049, U1736205, 61802166, 62102359, 62006181, 62132011, U20A20177, 62206217), State Key Laboratory of Communication Content Cognition (Grant No. A02103) and Shaanxi Province Key Industry Innovation Program (2023-ZDLGY-38, 2021ZDLGY01-02). Chao Shen is the corresponding author.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Jiang, W., Shen, C., Lin, C., Wang, J., Sun, J., Gao, X. (2023). Black-Box Fairness Testing with Shadow Models. In: Wang, D., Yung, M., Liu, Z., Chen, X. (eds) Information and Communications Security. ICICS 2023. Lecture Notes in Computer Science, vol 14252. Springer, Singapore. https://doi.org/10.1007/978-981-99-7356-9_28
Download citation
DOI: https://doi.org/10.1007/978-981-99-7356-9_28
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-7355-2
Online ISBN: 978-981-99-7356-9
eBook Packages: Computer ScienceComputer Science (R0)