Abstract
Self-taught learning models are successfully applied to improve the target model’s performance in different low-resource environments. In this setting, features are learned using unlabeled instances in the source domain; thereafter, the learned feature representations are transferred to the target domain for the supervised classification task. Two important challenges in this setup include learning efficient feature representations in the source domain and securing instance privacy against attacks carried out during knowledge transfer from the source to the target domain. We propose \(Meta-DPSTL\), a novel Meta Differentially Private Self-Taught Learning model to overcome these challenges. The proposed approach implements self-taught learning in the meta-learning-based framework; training of meta-learner and base-learner proceeds episodically and is equivalent to estimating source and target domain parameters, respectively. Further, to protect the sensitive source data from a potential attacker, differential privacy is added to the meta-parameters learned in an episode before they are passed to the target domain to train the base-learner. To measure the immunity of the proposed model to an inversion attack, we propose a novel Relative Reconstruction Distance (RRD) metric. Lastly, an inversion attack is carried out on the meta-parameters; empirical results obtained on the handwritten digits recognition dataset, \(COVID-19\) \(X-Ray\) Radiography dataset, and \(COVID-19\) Lung CT Scans dataset confirm the utility of meta-learning-based self-taught features in obtaining richer feature representations and hence, providing base-learners that are more generalizable. Relative reconstruction distance values computed on these datasets show that the differentially-private meta-parameters are robust to inversion attacks. Consequently, the proposed approach may be used in applications where the privacy requirements of sensitive source domain datasets are paramount.
Similar content being viewed by others
Data availability
The datasets in this study are not publicly available; however, the same will be made available upon reasonable request.
References
Adate A, Tripathy B (2022) A survey on deep learning methodologies of recent applications. Deep learning in data analytics. Springer, pp 145–170
Silva AJ, Cortez P, Pereira C, Pilastri A (2021) Business analytics in industry 4.0: a systematic review. Expert Syst 38(7):12741
Lauriola I, Lavelli A, Aiolli F (2022) An introduction to deep learning in natural language processing: models, techniques, and tools. Neurocomputing 470:443–456
Yang Q, Zhang Y, Dai W, Pan SJ (2020) Transfer learning. Cambridge University Press
Yu X, Wang J, Hong Q-Q, Teku R, Wang S-H, Zhang Y-D (2022) Transfer learning for medical images analyses: a survey. Neurocomputing 489:230–254
Alem A, Kumar S (2022) Transfer learning models for land cover and land use classification in remote sensing image. Appl Artif Intell 36(1):2014192
Song Y, Li J, Gao P, Li L, Tian T, Tian J (2022) Two-stage cross-modality transfer learning method for military-civilian sar ship recognition. IEEE Geosci Remote Sens Lett 19:1–5
Raina R, Battle A, Lee H, Packer B, Ng AY (2007) Self-taught learning: transfer learning from unlabeled data. In: Proceedings of the 24th International Conference on Machine Learning. pp 759–766
Xie L, Pan P, Lu Y, Jiang S (2015) Cross-modal self-taught learning for image retrieval. In: International Conference on Multimedia Modeling. Springer. pp 257–268
Sethi Y, Jain V, Singh KP, Ojha M (2017) Isomap based self-taught transfer learning for image classification. In: 2017 14th IEEE India Council International Conference (INDICON). IEEE. pp 1–6
Amft O (2011) Self-taught learning for activity spotting in on-body motion sensor data. In: 2011 15th Annual International Symposium on Wearable Computers. IEEE. pp 83–86
Jingjing H (2016) Self-taught learning features for human action recognition. In: 2016 3rd International Conference on Information Science and Control Engineering (ICISCE). IEEE. pp 611–615
Liu X (2019) Deep neural network based iterative self-taught learning on text mining. PhD thesis, University of Arkansas at Little Rock
Laghos A, Zaphiris P (2007) Social network analysis of self-taught e-learning communities. Int J Knowl Learn 3(4–5):465–482
Fayyaz M, Hajizadeh-Saffar M, Sabokrou M, Hoseini M, Fathy M (2015) A novel approach for finger vein verification based on self-taught learning. In: 2015 9th Iranian Conference on Machine Vision and Image Processing (MVIP). IEEE. pp 88–91
Zhu T, Philip SY (2019) Applying differential privacy mechanism in artificial intelligence. In: 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). IEEE. pp 1601–1609
Dankar FK, El Emam K (2012) The application of differential privacy to health data. In: Proceedings of the 2012 Joint EDBT/ICDT Workshops, pp 158–166
Abadi M, Chu A, Goodfellow I, McMahan HB, Mironov I, Talwar K, Zhang L (2016) Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp 308–318
Domingo-Ferrer J, Sánchez D, Blanco-Justicia A (2021) The limits of differential privacy (and its misuse in data release and machine learning). Commun ACM 64(7):33–35
Vu D, Slavkovic A (2009) Differential privacy for clinical trial data: Preliminary evaluations. In: 2009 IEEE International Conference on Data Mining Workshops. IEEE. pp 138–143
Vanschoren J (2018) Meta-learning: a survey. arXiv preprint arXiv:1810.03548
Gan J, Li L, Zhai Y, Liu Y (2014) Deep self-taught learning for facial beauty prediction. Neurocomputing 144:295–303
Zhou K, Liu Y, Song J, Yan L, Zou F, Shen F (2015) Deep self-taught hashing for image retrieval. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp 1215–1218
Jie Z, Wei Y, Jin X, Feng J, Liu W (2017) Deep self-taught learning for weakly supervised object localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1377–1385
Jie Z, Wei Y, Jin X, Feng J, Liu W (2017) Deep self-taught learning for weakly supervised object localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1377–1385
Xu J, Xu B, Wang P, Zheng S, Tian G, Zhao J (2017) Self-taught convolutional neural networks for short text clustering. Neural Netw 88:22–31
Zhang D, Wang J, Cai D, Lu J (2010) Self-taught hashing for fast similarity search. In: Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 18–25
Kuen J, Lim KM, Lee CP (2015) Self-taught learning of a deep invariant representation for visual tracking via temporal slowness principle. Pattern Recognit 48(10):2964–2982
Qureshi AS, Khan A, Shamim N, Durad MH (2020) Intrusion detection using deep sparse auto-encoder and self-taught learning. Neural Comput Appl 32(8):3135–3147
Roscher R, Römer C, Waske B, Plümer L (2015) Landcover classification with self-taught learning on archetypal dictionaries. In: 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS). IEEE. pp 2358–2361
Yang Y, Zhu D, Ren F, Cheng C (2020) A novel self-taught learning framework using spatial pyramid matching for scene classification. Int Arch Photogramm, Remote Sens Spatial Inf Sci 43:725–729
Bettge A, Roscher R, Wenzel S (2017) Deep self-taught learning for remote sensing image classification. arXiv preprint arXiv:1710.07096
Kemker R, Kanan C (2017) Self-taught feature learning for hyperspectral image classification. IEEE Trans Geosci Remote Sens 55(5):2693–2705
Roscher R, Wenzel S, Waske B (2016) Discriminative archetypal self-taught learning for multispectral landcover classification. In: 2016 9th IAPR Workshop on Pattern Recogniton in Remote Sensing (PRRS). IEEE. pp 1–5
Zhao Z, Zhang X, Chen C, Li W, Peng S, Wang J, Yang X, Zhang L, Zeng Z (2019) Semi-supervised self-taught deep learning for finger bones segmentation. In: 2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI). IEEE. pp 1–4
Hou C-A, Yang M-C, Wang Y-CF (2014) Domain adaptive self-taught learning for heterogeneous face recognition. In: 2014 22nd International Conference on Pattern Recognition. IEEE. pp 3068–3073
Hu H, Phan N, Geller J, Vo H, Manasi B, Huang X, Di Lorio S, Dinh T, Chun SA (2018) Deep self-taught learning for detecting drug abuse risk behavior in tweets. In: International Conference on Computational Social Networks. Springer. pp 330–342
He P, Jia P, Qiao S, Duan S (2017) Self-taught learning based on sparse autoencoder for e-nose in wound infection detection. Sensors 17(10):2279
Liu W, Ye X, Phan H, Hu H (2019) Scalable self-taught deep-embedded learning framework for drug abuse spatial behaviors detection. In: International Conference on Computational Data and Social Networks. Springer. pp 223–228
Huang K, Xu Z, King I, Lyu MR, Campbell C (2009) Supervised self-taught learning: Actively transferring knowledge from unlabeled data. In: 2009 International Joint Conference on Neural Networks. IEEE. pp 1272–1277
Fang M, Zhu X, Li B, Ding W, Wu X (2012) Self-taught active learning from crowds. In: 2012 IEEE 12th International Conference on Data Mining. IEEE. pp 858–863
Wang H, Nie F, Huang H (2013) Robust and discriminative self-taught learning. In: International Conference on Machine Learning. PMLR. pp 298–306
Zhang X, Liu Q, Wang D, Zhao L, Gu N, Maybank S (2019) Self-taught semisupervised dictionary learning with nonnegative constraint. IEEE Trans Ind Inform 16(1):532–543
Markov K, Matsui T (2012) Nonnegative matrix factorization based self-taught learning with application to music genre classification. In: 2012 IEEE International Workshop on Machine Learning for Signal Processing. IEEE. pp 1–5
Upadhyay R, Phlypo R, Saini R, Liwicki M (2021) Sharing to learn and learning to share-fitting together meta-learning, multi-task learning, and transfer learning: a meta review. arXiv preprint arXiv:2111.12146
Singh R, Bharti V, Purohit V, Kumar A, Singh AK, Singh SK (2021) Metamed: few-shot medical image classification using gradient-based meta-learning. Pattern Recognit 120:108111
Dumoulin V, Houlsby N, Evci U, Zhai X, Goroshin R, Gelly S, Larochelle H (2021) Comparing transfer and meta learning approaches on a unified few-shot classification benchmark. arXiv preprint arXiv:2104.02638
Huisman M, Van Rijn JN, Plaat A (2021) A survey of deep meta-learning. Artif Intell Rev 54(6):4483–4541
Zhou F, Cao C, Zhong T, Geng J (2021) Learning meta-knowledge for few-shot image emotion recognition. Expert Syst Appl 168:114274
Moniz N, Cerqueira V (2021) Automated imbalanced classification via meta-learning. Expert Syst Appl 178:115011
Dwork C (2008) Differential privacy: a survey of results. In: International Conference on Theory and Applications of Models of Computation. Springer. pp 1–19
Chowdhury ME, Rahman T, Khandakar A, Mazhar R, Kadir MA, Mahbub ZB, Islam KR, Khan MS, Iqbal A, Al Emadi N et al (2020) Can ai help in screening viral and COVID-19 pneumonia? IEEE Access 8:132665–132676
Dwork C, Karr A, Nissim K, Vilhuber L (2020) On privacy in the age of COVID-19. J Priv Confid. https://doi.org/10.29012/jpc.749
Machanavajjhala A, He X, Hay M (2017) Differential privacy in the wild: a tutorial on current practices & open challenges. In: Proceedings of the 2017 ACM International Conference on Management of Data, pp 1727–1730
Wang Q, Li Z, Zou Q, Zhao L, Wang S (2020) Deep domain adaptation with differential privacy. IEEE Trans Inf Forensics Secur 15:3093–3106
Hu R, Guo Y, Li H, Pei Q, Gong Y (2020) Personalized federated learning with differential privacy. IEEE Internet Things J 7(10):9530–9539
Ha T, Dang TK, Dang TT, Truong TA, Nguyen MT (2019) Differential privacy in deep learning: an overview. In: 2019 International Conference on Advanced Computing and Applications (ACOMP). IEEE. pp. 97–102
Dankar FK, El Emam K (2013) Practicing differential privacy in health care: a review. Trans Data Priv 6(1):35–67
Yang Y, Zhang Z, Miklau G, Winslett M, Xiao X (2012) Differential privacy in data publication and analysis. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp 601–606
Ji Z, Lipton ZC, Elkan C (2014) Differential privacy and machine learning: a survey and review. arXiv preprint arXiv:1412.7584
Ziller A, Usynin D, Braren R, Makowski M, Rueckert D, Kaissis G (2021) Medical imaging deep learning with differential privacy. Sci Rep 11(1):1–8
Ghaderzadeh M, Asadi F, Jafari R, Bashash D, Abolghasemi H, Aria M et al (2021) Deep convolutional neural network-based computer-aided detection system for COVID-19 using multiple lung scans: Design and implementation study. J Med Internet Res 23(4):27468
Hospedales T, Antoniou A, Micaelli P, Storkey A (2020) Meta-learning in neural networks: a survey. arXiv preprint arXiv:2004.05439
Chen S, Kahla M, Jia R, Qi G-J (2020) Knowledge-enriched distributional model inversion attacks. arXiv preprint arXiv:2010.04092
Park C, Hong D, Seo C (2019) An attack-based evaluation method for differentially private learning against model inversion attack. IEEE Access 7:124988–124999
Cohen G, Afshar S, Tapson J, Van Schaik A (2017) Emnist: extending mnist to handwritten letters. In: 2017 International Joint Conference on Neural Networks (IJCNN). IEEE. pp 2921–2926
Kermany D, Zhang K, Goldbaum M (2018) Labeled optical coherence tomography (oct) and chest x-ray images for classification. Mendeley Data 2(2):651
Abdiansah A, Wardoyo R (2015) Time complexity analysis of support vector machines (svm) in libsvm. Int J Comput Appl 128(3):28–34
Bulso N, Marsili M, Roudi Y (2019) On the complexity of logistic regression models. Neural Comput 31(8):1592–1623
Liu F, Ma J, Zhao R, Wang Q (2018) Online dictionary self-taught learning for hyperspectral image classification. In: 2018 IEEE International Instrumentation and Measurement Technology Conference (I2MTC). IEEE. pp 1–5
Chen X, Li B, Shamsabardeh M, Proietti R, Zhu Z, Yoo SB (2018) On real-time and self-taught anomaly detection in optical networks using hybrid unsupervised/supervised learning. In: 2018 European Conference on Optical Communication (ECOC). IEEE. pp 1–3
Bastien F, Bengio Y, Bergeron A, Boulanger-Lewandowski N, Breuel T, Chherawala Y, Cisse M, Côté M, Erhan D, Eustache J et al (2010) Deep self-taught learning for handwritten character recognition. arXiv preprint arXiv:1009.3589
Lee H, Raina R, Teichman A, Ng A (2009) Exponential family sparse coding with applications to self-taught learning. In: Twenty-First International Joint Conference on Artificial Intelligence
Chu K-F, Zhang L (2021) Privacy-preserving self-taught federated learning for heterogeneous data. arXiv preprint arXiv:2102.05883
Feng S, Yu H, Duarte MF (2020) Autoencoder based sample selection for self-taught learning. Knowl-Based Syst 192:105343
Razzaghi P (2019) Self-taught support vector machines. Knowl Inf Syst 59(3):685–709
Liu F, Dai S (2019) Tri-self-taught learning of artificial neural networks. In: Proceedings of 2018 Chinese Intelligent Systems Conference. Springer. pp 807–814
Le N (2019) Evolving self-taught neural networks: the Baldwin effect and the emergence of intelligence. arXiv preprint arXiv:1906.08854
Le N, Brabazon A, O’Neill M (2019) The evolution of self-taught neural networks in a multi-agent environment. In: International Conference on the Applications of Evolutionary Computation (Part of EvoStar). Springer. pp 457–472
Kamath U, Liu J, Whitaker J (2019) Transfer learning: Scenarios, self-taught learning, and multitask learning. In: Deep Learning for NLP and Speech Recognition. Springer. pp 463–493
Han X-H, Sun J, Lin L, Chen Y-W (2018) Self-taught learning with residual sparse autoencoders for hep-2 cell staining pattern recognition. In: International Workshop on Machine Learning in Medical Imaging. Springer. pp 134–142
Li S, Li K, Fu Y (2017) Self-taught low-rank coding for visual learning. IEEE Trans Neural Netw Learn Syst 29(3):645–656
Zhao F, Cao Z, Xiao Y, Zhang X, Xian K, Li R (2017) Depth image super-resolution via semi self-taught learning framework. In: Videometrics, Range Imaging, and Applications XIV, vol. 10332. International Society for Optics and Photonics. p 103320
Markov K, Matsui T (2013) High level feature extraction STL. EURASIP J Audio, Speech, Music Process 2013(1):1–11
Zhu X, Huang Z, Yang Y, Shen HT, Xu C, Luo J (2013) Self-taught dimensionality reduction on the high-dimensional small-sized data. Pattern Recognit 46(1):215–229
Dai W, Yang Q, Xue G-R, Yu Y (2008) Self-taught clustering. In: Proceedings of the 25th International Conference on Machine Learning, pp 200–207
El-Rashidy N, Abdelrazik S, Abuhmed T, Amer E, Ali F, Hu J-W, El-Sappagh S (2021) Comprehensive survey of using machine learning in the COVID-19 pandemic. Diagnostics 11(7):1155
Panday A, Kabir MA, Chowdhury NK (2021) A survey of machine learning techniques for detecting and diagnosing COVID-19 from imaging. arXiv preprint arXiv:2108.04344
Abir S, Islam SN, Anwar A, Mahmood AN, Oo AMT (2020) Building resilience against COVID-19 pandemic using artificial intelligence, machine learning, and iot: a survey of recent progress. IoT 1(2):506–528
Shahid O, Nasajpour M, Pouriyeh S, Parizi RM, Han M, Valero M, Li F, Aledhari M, Sheng QZ (2021) Machine learning research towards combating COVID-19: virus detection, spread prevention, and medical assistance. J Biomed Inform 117:103751
Kumar SA, Kumar H, Dutt V, Dixit P (2020) The role of machine learning in COVID-19 in medical domain: a survey. J Recent Innov Cloud Comput, Virtualiz Web Appl 4(1):1–12 (ISSN: 2581-544X (online))
John CC, Ponnusamy V, Chandrasekaran SK, Nandakumar R (2021) A survey on mathematical, machine learning and deep learning models for COVID-19 transmission and diagnosis. IEEE Rev Biomed Eng 15:325–40
Islam MN, Inan TT, Rafi S, Akter SS, Sarker IH, Islam A (2020) A survey on the use of ai and ml for fighting the COVID-19 pandemic. arXiv preprint arXiv:2008.07449
Sinha IK, Singh KP, Verma S (2020) Dp-ann: A new differential private artificial neural network with application on health data (workshop paper). In: 2020 IEEE Sixth International Conference on Multimedia Big Data (BigMM). IEEE. pp 351–355
Müftüoğlu Z, Kizrak MA, Yildlnm T (2020) Differential privacy practice on diagnosis of COVID-19 radiology imaging using efficientnet. In: 2020 International Conference on INnovations in Intelligent SysTems and Applications (INISTA). IEEE. pp 1–6
Bavadekar S, Dai A, Davis J, Desfontaines D, Eckstein I, Everett K, Fabrikant A, Flores G, Gabrilovich E, Gadepalli K et al (2020) Google COVID-19 search trends symptoms dataset: anonymization process description (version 1.0). arXiv preprint arXiv:2009.01265
Ulhaq A, Burmeister O (2020) COVID-19 imaging data privacy by federated learning design: a theoretical framework. arXiv preprint arXiv:2010.06177
Triastcyn A, Faltings B (2020) Bayesian differential privacy for machine learning. In: International Conference on Machine Learning. PMLR. pp 9583–9592
Dwork C, Roth A et al (2014) The algorithmic foundations of differential privacy. Found Trends Theor Comput Sci 9(3–4):211–407
Mivule K, Turner C, Ji S-Y (2012) Towards a differential privacy and utility preserving machine learning classifier. Procedia Comput Sci 12:176–181
Bagdasaryan E, Poursaeed O, Shmatikov V (2019) Differential privacy has disparate impact on model accuracy. Adv Neural Inf Process Syst 32:15479–15488
McMahan HB, Andrew G, Erlingsson U, Chien S, Mironov I, Papernot N, Kairouz P (2018) A general approach to adding differential privacy to iterative training procedures. arXiv preprint arXiv:1812.06210
Arachchige PCM, Bertok P, Khalil I, Liu D, Camtepe S, Atiquzzaman M (2019) Local differential privacy for deep learning. IEEE Internet Things J 7(7):5827–5842
Sun Z, Wang Y, Shu M, Liu R, Zhao H (2019) Differential privacy for data and model publishing of medical data. IEEE Access 7:152103–152114
Li N, Lyu M, Su D, Yang W (2016) Differential privacy: from theory to practice. Synth Lect Inf Secur, Priv, Trust 8(4):1–138
Zhu T, Li G, Zhou W, Philip SY (2017) Differential privacy and applications. Springer
Ficek J, Wang W, Chen H, Dagne G, Daley E (2021) Differential privacy in health research: a scoping review. J Am Med Inform Assoc 28(10):2269–2276
Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359
Acknowledgements
We acknowledge the Ministry of Education, Government of India, and Indian Institute of Information Technology Allahabad for providing infrastructure and financial support for this research.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All authors declare that there is no Conflict of interest in the subject matter discussed in this manuscript.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Singh, U.P., Sinha, I.K., Singh, K.P. et al. Meta-DPSTL: meta learning-based differentially private self-taught learning. Int. J. Mach. Learn. & Cyber. (2024). https://doi.org/10.1007/s13042-024-02134-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s13042-024-02134-2