Skip to main content

Differentially Private Hypothesis Transfer Learning

Part of the Lecture Notes in Computer Science book series (LNAI,volume 11052)

Abstract

In recent years, the focus of machine learning has been shifting to the paradigm of transfer learning where the data distribution in the target domain differs from that in the source domain. This is a prevalent setting in real-world classification problems and numerous well-established theoretical results in the classical supervised learning paradigm will break down under this setting. In addition, the increasing privacy protection awareness restricts access to source domain samples and poses new challenges for the development of privacy-preserving transfer learning algorithms. In this paper, we propose a novel differentially private multiple-source hypothesis transfer learning method for logistic regression. The target learner operates on differentially private hypotheses and importance weighting information from the sources to construct informative Gaussian priors for its logistic regression model. By leveraging a publicly available auxiliary data set, the importance weighting information can be used to determine the relationship between the source domain and the target domain without leaking source data privacy. Our approach provides a robust performance boost even when high quality labeled samples are extremely scarce in the target data set. The extensive experiments on two real-world data sets confirm the performance improvement of our approach over several baselines. Data related to this paper is available at: http://qwone.com/~jason/20Newsgroups/ and https://www.cs.jhu.edu/~mdredze/datasets/sentiment/index2.html.

Keywords

  • Differential privacy
  • Transfer learning

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-10928-8_48
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   99.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-10928-8
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   129.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.

Notes

  1. 1.

    http://qwone.com/~jason/20Newsgroups/.

  2. 2.

    https://www.cs.jhu.edu/~mdredze/datasets/sentiment/index2.html.

References

  1. Blitzer, J., Dredze, M., Pereira, F.: Biographies, bollywood, boom-boxes and blenders: domain adaptation for sentiment classification. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 440–447 (2007)

    Google Scholar 

  2. Boker, S.M., et al.: Maintained individual data distributed likelihood estimation (middle). Multivar. Behav. Res. 50(6), 706–720 (2015)

    CrossRef  Google Scholar 

  3. Chaudhuri, K., Monteleoni, C., Sarwate, A.D.: Differentially private empirical risk minimization. J. Mach. Learn. Res. 12, 1069–1109 (2011)

    MathSciNet  MATH  Google Scholar 

  4. Dwork, C., McSherry, F., Nissim, K., Smith, A.: Calibrating noise to sensitivity in private data analysis. In: Halevi, S., Rabin, T. (eds.) TCC 2006. LNCS, vol. 3876, pp. 265–284. Springer, Heidelberg (2006). https://doi.org/10.1007/11681878_14

    CrossRef  Google Scholar 

  5. Dwork, C., Roth, A.: The algorithmic foundations of differential privacy. Found. Trends Theoret. Comput. Sci. 9(3–4), 211–407 (2014)

    MathSciNet  MATH  Google Scholar 

  6. Garcke, J., Vanck, T.: Importance weighted inductive transfer learning for regression. In: Calders, T., Esposito, F., Hüllermeier, E., Meo, R. (eds.) ECML PKDD 2014. LNCS (LNAI), vol. 8724, pp. 466–481. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-44848-9_30

    CrossRef  Google Scholar 

  7. Gupta, S.K., Rana, S., Venkatesh, S.: Differentially private multi-task learning. In: Chau, M., Wang, G.A., Chen, H. (eds.) PAISI 2016. LNCS, vol. 9650, pp. 101–113. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-31863-9_8

    CrossRef  Google Scholar 

  8. Hamm, J., Cao, Y., Belkin, M.: Learning privately from multiparty data. In: International Conference on Machine Learning, pp. 555–563 (2016)

    Google Scholar 

  9. Ji, Z., Elkan, C.: Differential privacy based on importance weighting. Mach. Learn. 93(1), 163–183 (2013)

    MathSciNet  CrossRef  Google Scholar 

  10. Kuzborskij, I., Orabona, F.: Stability and hypothesis transfer learning. In: Proceedings of The 30th International Conference on Machine Learning, pp. 942–950. ACM (2013)

    Google Scholar 

  11. Lang, K.: Newsweeder: learning to filter netnews. In: Machine Learning Proceedings 1995, pp. 331–339. Elsevier (1995)

    Google Scholar 

  12. Mansour, Y., Mohri, M., Rostamizadeh, A.: Domain adaptation with multiple sources. In: Advances in Neural Information Processing Systems, pp. 1041–1048 (2009)

    Google Scholar 

  13. Marx, Z., Rosenstein, M.T., Dietterich, T.G., Kaelbling, L.P.: Two algorithms for transfer learning. Inductive transfer: 10 years later (2008)

    Google Scholar 

  14. Mihalkova, L., Mooney, R.J.: Transfer learning by mapping with minimal target data. In: Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI) Workshop on Transfer Learning for Complex Tasks (2008)

    Google Scholar 

  15. Nedic, A., Ozdaglar, A.: Distributed subgradient methods for multi-agent optimization. IEEE Trans. Autom. Control 54(1), 48–61 (2009)

    MathSciNet  CrossRef  Google Scholar 

  16. Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)

    CrossRef  Google Scholar 

  17. Pan, W., Zhong, E., Yang, Q.: Transfer learning for text mining. In: Aggarwal, C., Zhai, C. (eds.) Mining Text Data, pp. 223–257. Springer, Boston (2012)

    CrossRef  Google Scholar 

  18. Papernot, N., Song, S., Mironov, I., Raghunathan, A., Talwar, K., Erlingsson, U.: Scalable private learning with PATE. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=rkZB1XbRZ

  19. Pathak, M., Rane, S., Raj, B.: Multiparty differential privacy via aggregation of locally trained classifiers. In: Advances in Neural Information Processing Systems, pp. 1876–1884 (2010)

    Google Scholar 

  20. Raina, R., Battle, A., Lee, H., Packer, B., Ng, A.Y.: Self-taught learning: transfer learning from unlabeled data. In: Proceedings of the 24th International Conference on Machine Learning, pp. 759–766. ACM (2007)

    Google Scholar 

  21. Raina, R., Ng, A.Y., Koller, D.: Constructing informative priors using transfer learning. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 713–720. ACM (2006)

    Google Scholar 

  22. Rosenstein, M.T., Marx, Z., Kaelbling, L.P., Dietterich, T.G.: To transfer or not to transfer. In: NIPS 2005 Workshop on Transfer Learning, vol. 898, pp. 1–4 (2005)

    Google Scholar 

  23. Shokri, R., Shmatikov, V.: Privacy-preserving deep learning. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1310–1321. ACM (2015)

    Google Scholar 

  24. Wang, Y., et al.: Privacy preserving distributed deep learning and its application in credit card fraud detection. In: 2018 17th IEEE International Conference on Trust, Security and Privacy In Computing And Communications/12th IEEE International Conference on Big Data Science and Engineering (TrustCom/BigDataSE), pp. 1070–1078. IEEE (2018)

    Google Scholar 

  25. Xie, L., Baytas, I.M., Lin, K., Zhou, J.: Privacy-preserving distributed multi-task learning with asynchronous updates. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1195–1204. ACM (2017)

    Google Scholar 

Download references

Acknowledgements

We would like to thank the anonymous reviewers for their helpful comments. This work is partially supported by a grant from the Army Research Laboratory W911NF-17-2-0110. Quanquan Gu is partly supported by the National Science Foundation SaTC CNS-1717206. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yang Wang .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Wang, Y., Gu, Q., Brown, D. (2019). Differentially Private Hypothesis Transfer Learning. In: Berlingerio, M., Bonchi, F., Gärtner, T., Hurley, N., Ifrim, G. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2018. Lecture Notes in Computer Science(), vol 11052. Springer, Cham. https://doi.org/10.1007/978-3-030-10928-8_48

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-10928-8_48

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-10927-1

  • Online ISBN: 978-3-030-10928-8

  • eBook Packages: Computer ScienceComputer Science (R0)