Abstract
Privacy-preserving training of machine learning models aims to avoid or minimize (mitigate) the exact or similar reproduction (leakage) of information contained in the training data. This chapter introduces pre-processing methods (filtering and de-duplication) that prepare the training data to minimize information leakage, followed by a discussion of training and deployment methods (differentially private fine-tuning, noisy knowledge transfer) that provide empirical or theoretical guarantees for the achieved privacy protection with a focus on Large Language Models (LLMs).
Chapter PDF
References
Katherine Lee et al. Deduplicating training data makes language models better, 2022.
Florian Tramèr et al. Truth serum: Poisoning machine learning models to reveal their secrets, 2022.
Arvind Narayanan and Vitaly Shmatikov. How to break anonymity of the netflix prize dataset, 2007.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211–407, 2014.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models, 2018.
Gavin Kerrigan, Dylan Slack, and Jens Tuyls. Differentially private language models benefit from public pre-training, 2020.
Nicolas Papernot et al. Making the shoe fit: Architectures, initializations, and tuning for learning with privacy, 2020.
Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners, 2022.
Da Yu et al. Differentially private fine-tuning of language models, 2022.
Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of stochastic parrots: Differentially private prompt learning for large language models, 2023.
Nicolas Papernot et al. Scalable private learning with pate, 2018.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2024 The Author(s)
About this chapter
Cite this chapter
Buesser, B. (2024). Towards Privacy Preserving LLMs Training. In: Kucharavy, A., Plancherel, O., Mulder, V., Mermoud, A., Lenders, V. (eds) Large Language Models in Cybersecurity. Springer, Cham. https://doi.org/10.1007/978-3-031-54827-7_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-54827-7_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-54826-0
Online ISBN: 978-3-031-54827-7
eBook Packages: Computer ScienceComputer Science (R0)