Abstract
Fairness testing, given a machine learning classifier, detects discriminatory data contained in it via executing test cases. In this paper, we propose a new approach to fairness testing named Vbt-Ct, which applies combinatorial t-way testing (CT) to Verification Based Testing (Vbt). Vbt is a state-of-the-art fairness testing method, which represents a given classifier under test in logical constraints and searches for test cases by solving such constraints. CT is a coverage-based sampling technique, with an ability to sample diverse test data from a search space specified by logical constraints. We implement a proof-of-concept of Vbt-Ct, and see its feasibility by experiments. We also discuss its advantages, current limitations, and further research directions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
We do not find their algorithm implementation is publicly available.
References
Aggarwal, A., Lohia, P., Nagar, S., Dey, K., Saha, D.: Black box fairness testing of machine learning models. In: Proceedings of ESEC/SIGSOFT FSE, pp. 625–635 (2019). https://doi.org/10.1145/3338906.3338937
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of ITCS 2012, pp. 214–226 (2012)
Galhotra, S., Brun, Y., Meliou, A.: Fairness testing: testing software for discrimination. In: Proceedings of ESEC/FSE 2017, pp. 498–510. ACM (2017)
Angwin, J., Larson, J., Mattu, S., Kirchner, L.: Machine bias (2016). https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
Kuhn, R.: Automated combinatorial testing for software (2016). https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software. Accessed 24 Feb 2021
Kuhn, R., Kacker, R.: Introduction to Combinatorial Testing. Chapman & Hall CRC (2013)
Microsoft: Pairwise independent combinatorial testing. https://github.com/microsoft/pict
Morales, D.P., Kitamura, T., Takada, S.: Coverage-guided fairness testing. In: Proceedings of ICIS 2021, pp. 183–199 (2021)
Patel, A.R., Chandrasekaran, J., Lei, Y., Kacker, R.N., Kuhn, D.R.: A combinatorial approach to fairness testing of machine learning models. In: Proceedings of IWCT 2022, pp. 1135–1144. IEEE (2022)
Sharma, A., Demir, C., Ngomo, A.N., Wehrheim, H.: MLCHECK-property-driven testing of machine learning classifiers. In: Proceedings of ICMLA 2021, pp. 738–745. IEEE (2021)
Sharma, A., Wehrheim, H.: Automatic fairness testing of machine learning models. In: Casola, V., De Benedictis, A., Rak, M. (eds.) ICTSS 2020. LNCS, vol. 12543, pp. 255–271. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64881-7_16
Udeshi, S., Arora, P., Chattopadhyay, S.: Automated directed fairness testing. In: Proceedings of ASE 2018, pp. 98–108 (2018)
Yamada, A., Biere, A., Artho, C., Kitamura, T., Choi, E.: Greedy combinatorial test case generation using unsatisfiable cores. In: Proceedings of ASE 2016, pp. 614–624. ACM (2016)
Zhang, P., et al.: White-box fairness testing through adversarial sampling. In: Proceedings of ICSE 2020, pp. 949–960. ACM (2020)
Acknowledgements
This paper is partly based on results obtained from a project, JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kitamura, T., Zhao, Z., Toda, T. (2022). Applying Combinatorial Testing to Verification-Based Fairness Testing. In: Papadakis, M., Vergilio, S.R. (eds) Search-Based Software Engineering. SSBSE 2022. Lecture Notes in Computer Science, vol 13711. Springer, Cham. https://doi.org/10.1007/978-3-031-21251-2_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-21251-2_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-21250-5
Online ISBN: 978-3-031-21251-2
eBook Packages: Computer ScienceComputer Science (R0)