Advertisement

Fast Homomorphic Evaluation of Deep Discretized Neural Networks

  • Florian Bourse
  • Michele Minelli
  • Matthias Minihold
  • Pascal Paillier
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10993)

Abstract

The rise of machine learning as a service multiplies scenarios where one faces a privacy dilemma: either sensitive user data must be revealed to the entity that evaluates the cognitive model (e.g., in the Cloud), or the model itself must be revealed to the user so that the evaluation can take place locally. Fully Homomorphic Encryption (FHE) offers an elegant way to reconcile these conflicting interests in the Cloud-based scenario and also preserve non-interactivity. However, due to the inefficiency of existing FHE schemes, most applications prefer to use Somewhat Homomorphic Encryption (SHE), where the complexity of the computation to be performed has to be known in advance, and the efficiency of the scheme depends on this global complexity.

In this paper, we present a new framework for homomorphic evaluation of neural networks, that we call FHE–DiNN, whose complexity is strictly linear in the depth of the network and whose parameters can be set beforehand. To obtain this scale-invariance property, we rely heavily on the bootstrapping procedure. We refine the recent FHE construction by Chillotti et al. (ASIACRYPT 2016) in order to increase the message space and apply the sign function (that we use to activate the neurons in the network) during the bootstrapping. We derive some empirical results, using TFHE library as a starting point, and classify encrypted images from the MNIST dataset with more than 96% accuracy in less than 1.7 s.

Finally, as a side contribution, we analyze and introduce some variations to the bootstrapping technique of Chillotti et al. that offer an improvement in efficiency at the cost of increasing the storage requirements.

Keywords

Fully homomorphic encryption Neural networks Bootstrapping MNIST 

Notes

Acknowledgments

Florian Bourse was supported by the European Research Council under the European Community’s Seventh Framework Programme (FP7/2007-2013 Grant Agreement no. 339563 – CryptoCloud), and by the French ANR Project ANR-16-CE39-0014 PERSOCLOUD. Part of this work was done while the author was employed by CNRS and visiting CryptoExperts.

Michele Minelli and Matthias Minihold were supported by European Union’s Horizon 2020 research and innovation programme under grant agreement No H2020-MSCA-ITN-2014-643161 ECRYPT-NET. This work was done while the authors were visiting CryptoExperts. The authors would like to thank CRYPTO’s anonymous reviewers for providing useful suggestions and helping improve the paper.

References

  1. [AAB+15]
    Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). Software: tensorflow.org
  2. [ABDP15]
    Abdalla, M., Bourse, F., De Caro, A., Pointcheval, D.: Simple functional encryption schemes for inner products. In: Katz, J. (ed.) PKC 2015. LNCS, vol. 9020, pp. 733–751. Springer, Heidelberg (2015).  https://doi.org/10.1007/978-3-662-46447-2_33CrossRefGoogle Scholar
  3. [ALS16]
    Agrawal, S., Libert, B., Stehlé, D.: Fully secure functional encryption for inner products, from standard assumptions. In: Robshaw, M., Katz, J. (eds.) CRYPTO 2016, Part III. LNCS, vol. 9816, pp. 333–362. Springer, Heidelberg (2016).  https://doi.org/10.1007/978-3-662-53015-3_12CrossRefGoogle Scholar
  4. [AP13]
    Alperin-Sheriff, J., Peikert, C.: Practical bootstrapping in quasilinear time. In: Canetti, R., Garay, J.A. (eds.) CRYPTO 2013, Part I. LNCS, vol. 8042, pp. 1–20. Springer, Heidelberg (2013).  https://doi.org/10.1007/978-3-642-40041-4_1CrossRefzbMATHGoogle Scholar
  5. [AP14]
    Alperin-Sheriff, J., Peikert, C.: Faster bootstrapping with polynomial error. In: Garay, J.A., Gennaro, R. (eds.) CRYPTO 2014, Part I. LNCS, vol. 8616, pp. 297–314. Springer, Heidelberg (2014).  https://doi.org/10.1007/978-3-662-44371-2_17CrossRefGoogle Scholar
  6. [APS15]
    Albrecht, M.R., Player, R., Scott, S.: On the concrete hardness of learning with errors. Cryptology ePrint Archive, Report 2015/046 (2015). http://eprint.iacr.org/2015/046
  7. [AS00]
    Agrawal, R., Srikant, R.: Privacy-preserving data mining. SIGMOD Rec. 29(2), 439–450 (2000)CrossRefGoogle Scholar
  8. [BGV12]
    Brakerski, Z., Gentry, C., Vaikuntanathan, V.: (Leveled) fully homomorphic encryption without bootstrapping. In: ITCS 2012, pp. 309–325. ACM, January 2012Google Scholar
  9. [BLMZ17]
    Benhamouda, F., Lepoint, T., Mathieu, C., Zhou, H.: Optimization of bootstrapping in circuits. In: Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2017, Philadelphia, PA, USA, pp. 2423–2433. Society for Industrial and Applied Mathematics (2017)Google Scholar
  10. [BPTG15]
    Bost, R., Popa, R.A., Tu, S., Goldwasser, S.: Machine learning classification over encrypted data. In: NDSS 2015. The Internet Society, February 2015Google Scholar
  11. [BV11a]
    Brakerski, Z., Vaikuntanathan, V.: Efficient fully homomorphic encryption from (standard) LWE. In: 52nd FOCS, pp. 97–106. IEEE Computer Society Press, October 2011Google Scholar
  12. [BV11b]
    Brakerski, Z., Vaikuntanathan, V.: Fully homomorphic encryption from ring-LWE and security for key dependent messages. In: Rogaway, P. (ed.) CRYPTO 2011. LNCS, vol. 6841, pp. 505–524. Springer, Heidelberg (2011).  https://doi.org/10.1007/978-3-642-22792-9_29CrossRefGoogle Scholar
  13. [BV14]
    Brakerski, Z., Vaikuntanathan, V.: Lattice-based FHE as secure as PKE. In: ITCS 2014, pp. 1–12. ACM, January 2014Google Scholar
  14. [C+15]
    Chollet, F., et al.: Keras (2015). https://github.com/keras-team/keras
  15. [CB16]
    Courbariaux, M., Bengio, Y.: Binarynet: training deep neural networks with weights and activations constrained to +1 or \(-1\). CoRR, abs/1602.02830 (2016)Google Scholar
  16. [CdWM+17]
    Chabanne, H., de Wargny, A., Milgram, J., Morel, C., Prouff, E.: Privacy-preserving classification on deep neural network. IACR Cryptology ePrint Archive 2017:35 (2017)Google Scholar
  17. [CGGI16a]
    Chillotti, I., Gama, N., Georgieva, M., Izabachène, M.: TFHE: Fast Fully Homomorphic Encryption Library over the Torus (2016). https://github.com/tfhe/tfhe
  18. [CGGI16b]
    Chillotti, I., Gama, N., Georgieva, M., Izabachène, M.: Faster fully homomorphic encryption: bootstrapping in less than 0.1 seconds. In: Cheon, J.H., Takagi, T. (eds.) ASIACRYPT 2016, Part I. LNCS, vol. 10031, pp. 3–33. Springer, Heidelberg (2016).  https://doi.org/10.1007/978-3-662-53887-6_1CrossRefzbMATHGoogle Scholar
  19. [CGGI17]
    Chillotti, I., Gama, N., Georgieva, M., Izabachène, M.: Faster packed homomorphic operations and efficient circuit bootstrapping for TFHE. In: Takagi, T., Peyrin, T. (eds.) ASIACRYPT 2017, Part I. LNCS, vol. 10624, pp. 377–408. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-70694-8_14CrossRefGoogle Scholar
  20. [CMS12]
    Cireşan, D., Meier, U., Schmidhuber, J.: Multi-column deep neural networks for image classification. ArXiv e-prints, February 2012Google Scholar
  21. [Cyb89]
    Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Sig. Syst. 2(4), 303–314 (1989)MathSciNetCrossRefGoogle Scholar
  22. [DGBL+16]
    Dowlin, N., Gilad-Bachrach, R., Laine, K., Lauter, K., Naehrig, M., Wernsing, J.: CryptoNets: applying neural networks to encrypted data with high throughput and accuracy. Technical report, February 2016Google Scholar
  23. [DM15]
    Ducas, L., Micciancio, D.: FHEW: bootstrapping homomorphic encryption in less than a second. In: Oswald, E., Fischlin, M. (eds.) EUROCRYPT 2015, Part I. LNCS, vol. 9056, pp. 617–640. Springer, Heidelberg (2015).  https://doi.org/10.1007/978-3-662-46800-5_24CrossRefzbMATHGoogle Scholar
  24. [Dwo06]
    Dwork, C.: Differential privacy (invited paper). In: Bugliesi, M., Preneel, B., Sassone, V., Wegener, I. (eds.) ICALP 2006, Part II. LNCS, vol. 4052, pp. 1–12. Springer, Heidelberg (2006).  https://doi.org/10.1007/11787006_1CrossRefGoogle Scholar
  25. [Gen09]
    Gentry, C.: A fully homomorphic encryption scheme. Ph.D. thesis, Stanford University (2009). crypto.stanford.edu/craig
  26. [GHS12]
    Gentry, C., Halevi, S., Smart, N.P.: Homomorphic evaluation of the AES circuit. In: Safavi-Naini, R., Canetti, R. (eds.) CRYPTO 2012. LNCS, vol. 7417, pp. 850–867. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-32009-5_49CrossRefGoogle Scholar
  27. [GSW13]
    Gentry, C., Sahai, A., Waters, B.: Homomorphic encryption from learning with errors: conceptually-simpler, asymptotically-faster, attribute-based. In: Canetti, R., Garay, J.A. (eds.) CRYPTO 2013, Part I. LNCS, vol. 8042, pp. 75–92. Springer, Heidelberg (2013).  https://doi.org/10.1007/978-3-642-40041-4_5CrossRefGoogle Scholar
  28. [Hor91]
    Hornik, K.: Approximation capabilities of multilayer feedforward networks. Neural Netw. 4(2), 251–257 (1991)MathSciNetCrossRefGoogle Scholar
  29. [HS14]
    Halevi, S., Shoup, V.: Algorithms in HElib. In: Garay, J.A., Gennaro, R. (eds.) CRYPTO 2014, Part I. LNCS, vol. 8616, pp. 554–571. Springer, Heidelberg (2014).  https://doi.org/10.1007/978-3-662-44371-2_31CrossRefzbMATHGoogle Scholar
  30. [HS15]
    Halevi, S., Shoup, V.: Bootstrapping for HElib. In: Oswald, E., Fischlin, M. (eds.) EUROCRYPT 2015, Part I. LNCS, vol. 9056, pp. 641–670. Springer, Heidelberg (2015).  https://doi.org/10.1007/978-3-662-46800-5_25CrossRefGoogle Scholar
  31. [HZRS15]
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR, abs/1512.03385 (2015)Google Scholar
  32. [KTX08]
    Kawachi, A., Tanaka, K., Xagawa, K.: Concurrently secure identification schemes based on the worst-case hardness of lattice problems. In: Pieprzyk, J. (ed.) ASIACRYPT 2008. LNCS, vol. 5350, pp. 372–389. Springer, Heidelberg (2008).  https://doi.org/10.1007/978-3-540-89255-7_23CrossRefGoogle Scholar
  33. [LBBH98]
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  34. [LCB98]
    LeCun, Y., Cortes, C., Burges, C.: The MNIST database of handwritten digits (1998). http://yann.lecun.com/exdb/mnist/
  35. [LP13]
    Lepoint, T., Paillier, P.: On the minimal number of bootstrappings in homomorphic circuits. In: Adams, A.A., Brenner, M., Smith, M. (eds.) FC 2013 Workshops. LNCS, vol. 7862, pp. 189–200. Springer, Heidelberg (2013).  https://doi.org/10.1007/978-3-642-41320-9_13CrossRefGoogle Scholar
  36. [LPR10]
    Lyubashevsky, V., Peikert, C., Regev, O.: On ideal lattices and learning with errors over rings. In: Gilbert, H. (ed.) EUROCRYPT 2010. LNCS, vol. 6110, pp. 1–23. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-13190-5_1CrossRefGoogle Scholar
  37. [MRSV17]
    Makri, E., Rotaru, D., Smart, N.P., Vercauteren, F.: PICS: private image classification with SVM. Cryptology ePrint Archive, Report 2017/1190 (2017). https://eprint.iacr.org/2017/1190
  38. [MZ17]
    Mohassel, P., Zhang, Y.: SecureML: a system for scalable privacy-preserving machine learning. In: 2017 IEEE Symposium on Security and Privacy, pp. 19–38. IEEE Computer Society Press, May 2017Google Scholar
  39. [PV16]
    Paindavoine, M., Vialla, B.: Minimizing the number of bootstrappings in fully homomorphic encryption. In: Dunkelman, O., Keliher, L. (eds.) SAC 2015. LNCS, vol. 9566, pp. 25–43. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-31301-6_2CrossRefzbMATHGoogle Scholar
  40. [PW08]
    Peikert, C., Waters, B.: Lossy trapdoor functions and their applications. In 40th ACM STOC, pp. 187–196. ACM Press, May 2008Google Scholar
  41. [Reg05]
    Regev, O.: On lattices, learning with errors, random linear codes, and cryptography. In: 37th ACM STOC, pp. 84–93. ACM Press, May 2005Google Scholar
  42. [SS10]
    Stehlé, D., Steinfeld, R.: Faster fully homomorphic encryption. In: Abe, M. (ed.) ASIACRYPT 2010. LNCS, vol. 6477, pp. 377–394. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-17373-8_22CrossRefGoogle Scholar
  43. [SV10]
    Smart, N.P., Vercauteren, F.: Fully homomorphic encryption with relatively small key and ciphertext sizes. In: Nguyen, P.Q., Pointcheval, D. (eds.) PKC 2010. LNCS, vol. 6056, pp. 420–443. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-13013-7_25CrossRefzbMATHGoogle Scholar
  44. [vDGHV10]
    van Dijk, M., Gentry, C., Halevi, S., Vaikuntanathan, V.: Fully homomorphic encryption over the integers. In: Gilbert, H. (ed.) EUROCRYPT 2010. LNCS, vol. 6110, pp. 24–43. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-13190-5_2CrossRefGoogle Scholar
  45. [ZK16]
    Zagoruyko, S., Komodakis, N.: Wide residual networks. CoRR, abs/1605.07146 (2016)Google Scholar
  46. [ZYC16]
    Zhang, Q., Yang, L.T., Chen, Z.: Privacy preserving deep computation model on cloud for big data feature learning. IEEE Trans. Comput. 65(5), 1351–1362 (2016)MathSciNetCrossRefGoogle Scholar
  47. [ZYL+17]
    Zhou, T., Yang, X., Liu, L., Zhang, W., Ding, Y.: Faster bootstrapping with multiple addends. Cryptology ePrint Archive, Report 2017/735 (2017). http://eprint.iacr.org/2017/735

Copyright information

© International Association for Cryptologic Research 2018

Authors and Affiliations

  • Florian Bourse
    • 1
  • Michele Minelli
    • 2
    • 3
  • Matthias Minihold
    • 4
  • Pascal Paillier
    • 5
  1. 1.Orange Labs, Applied Crypto GroupCesson-SévignéFrance
  2. 2.DIENS, École normale supérieure, CNRS, PSL Research UniversityParisFrance
  3. 3.InriaParisFrance
  4. 4.Horst Görtz Institut für IT-Security, Ruhr-Universität BochumBochumGermany
  5. 5.CryptoExpertsParisFrance

Personalised recommendations