Skip to main content
SpringerLink
Log in

Robust model selection for positive and unlabeled learning with constraints

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Positive and unlabeled (PU) learning is the problem in which training data contains only PU samples. Although PU learning is widely used in real-world applications, its model selection remains challenging. Specifically, traditional model selection methods are often highly sensitive to the class prior as well as the data size, resulting in human overhead in hyperparameter optimization. In this paper, we present a method called ODE (robust model selection) for robust model selection in PU learning. Two novel model evaluators based on the integral probability metric and area under the curve, which are free of the class prior, are introduced, and a variance reduction method is further employed to improve the quality of model selection. In addition, we perform model selection under user-defined constraints and propose a fast halving-style searching algorithm to efficiently identify the most promising model configuration. Extensive empirical studies demonstrate that our proposed method performs more robustly and is more computationally efficient than many state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Hsieh C, Natarajan N, Dhillon I S. PU learning for matrix completion. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning, 2015. 2445–2453

  2. McAuley J, Pandey R, Leskovec J. Inferring networks of substitutable and complementary products. In: Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015. 785–794

  3. Partalas I, Kosmopoulos A, Baskiotis N, et al. LSHTC: a benchmark for large-scale text classification. 2015. ArXiv: 1503.08581

  4. Wei H H, Li M. Positive and unlabeled learning for detecting software functional clones with adversarial training. In: Proceedings of International Joint Conference on Artificial Intelligence, 2018. 2840–2846

  5. Nguyen M N, Li X L, Ng S K. Positive unlabeled leaning for time series classification. In: Proceedings of the 22nd International Joint Conference on Artificial Intelligence, 2011. 1421–1426

  6. Yang P, Li X L, Chua H N, et al. Ensemble positive unlabeled learning for disease gene identification. PLoS ONE, 2014, 9: 97079

    Article  Google Scholar 

  7. Liu B, Lee W S, Yu P S, et al. Partially supervised classification of text documents. In: Proceedings of the 19th International Conference on Machine Learning, 2002. 387–394

  8. Liu B, Dai Y, Li X L, et al. Building text classifiers using positive and unlabeled examples. In: Proceedings of the 3rd IEEE International Conference on Data Mining ICDM, 2003. 179–188

  9. Li X L, Liu B. Learning to classify texts using positive and unlabeled data. In: Proceedings of International Joint Conference on Artificial Intelligence, 2003. 587–592

  10. Wei T, Shi F, Wang H, et al. MixPUL: consistency-based augmentation for positive and unlabeled learning. 2020. ArXiv:2004.09388

  11. Lee W S, Liu B. Learning with positive and unlabeled examples using weighted logistic regression. In: Proceedings of the 20th International Conference on Machine Learning, 2003. 448–455

  12. Shi H, Pan S J, Yang J, et al. Positive and unlabeled learning via loss decomposition and centroid estimation. In: Proceedings of International Joint Conference on Artificial Intelligence, 2018. 2689–2695

  13. Elkan C, Noto K. Learning classifiers from only positive and unlabeled data. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2008. 213–220

  14. du Plessis M C, Niu G, Sugiyama M. Analysis of learning from positive and unlabeled data. In: Proceedings of Annual Conference on Neural Information Processing Systems, 2014. 703–711

  15. du Plessis M C, Niu G, Sugiyama M. Convex formulation for learning from positive and unlabeled data. In: Proceedings of the 32nd International Conference on Machine Learning, 2015. 1386–1394

  16. Kiryo R, Niu G, du Plessis M C, et al. Positive-unlabeled learning with non-negative risk estimator. In: Proceedings of Annual Conference on Neural Information Processing Systems, 2017. 1674–1684

  17. Thornton C, Hutter F, Hoos H H, et al. Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2013. 847–855

  18. Feurer M, Klein A, Eggensperger K, et al. Efficient and robust automated machine learning. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, 2015. 2962–2970

  19. Yao Q M, Wang M S, Hugo J E, et al. Taking human out of learning applications: a survey on automated machine learning. 2018. ArXiv:1810.13306

  20. Li Y F, Wang H, Wei T, et al. Towards automated semi-supervised learning. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence, 2019

  21. Wei T, Guo L Z, Li Y F, et al. Learning safe multi-label prediction for weakly labeled data. Mach Learn, 2018, 107: 703–725

    Article  MathSciNet  MATH  Google Scholar 

  22. Dietterich T G. Robust artificial intelligence and robust human organizations. Front Comput Sci, 2019, 13: 1–3

    Article  Google Scholar 

  23. Li Y F, Liang D M. Safe semi-supervised learning: a brief introduction. Front Comput Sci, 2019, 13: 669–676

    Article  Google Scholar 

  24. Zhou Z H. Abductive learning: towards bridging machine learning and logical reasoning. Sci China Inf Sci, 2019, 62: 076101

    Article  MathSciNet  Google Scholar 

  25. Tolstikhin I O, Bousquet O, Gelly S, et al. Wasserstein auto-encoders. In: Proceedings of the 6th International Conference on Learning Representations, 2018

  26. Xie Z, Li M. Semi-supervised AUC optimization without guessing labels of unlabeled data. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 2018. 4310–4317

  27. Niu G, du Plessis M C, Sakai T, et al. Theoretical comparisons of positive-unlabeled learning against positive-negative learning. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016. 1199–1207

  28. du Plessis M C, Niu G, Sugiyama M. Analysis of learning from positive and unlabeled data. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, 2014. 703–711

  29. Claesen M, Davis J, de Smet F, et al. Assessing binary classifiers using only positive and unlabeled data. 2015. ArXiv:1504.06837

  30. Jain S, White M, Radivojac P. Recovering true classifier performance in positive-unlabeled learning. In: Proceedings of the 31st AAAI Conference on Artificial Intelligence, 2017. 2066–2072

  31. Conn A R, Scheinberg K, Vicente L N. Introduction to Derivative-free Optimization. Philadelphia: SIAM, 2009

    Book  MATH  Google Scholar 

  32. Bergstra J, Bengio Y. Random search for hyper-parameter optimization. J Mach Learn Res, 2012, 13: 281–305

    MathSciNet  MATH  Google Scholar 

  33. Bergstra J S, Bardenet R, Bengio Y, et al. Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems, 2011. 2546–2554

  34. Snoek J, Larochelle H, Adams R P. Practical bayesian optimization of machine learning algorithms. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, 2012. 2951–2959

  35. Klein A, Falkner S, Bartels S, et al. Fast bayesian optimization of machine learning hyperparameters onlarge datasets. 2016. ArXiv:1605.07079

  36. Bekker J, Davis J. Estimating the class prior in positive and unlabeled data through decision tree induction. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 2018. 2712–2719

  37. Menon A K, van Rooyen B, Ong C S, et al. Learning from corrupted binary labels via class-probability estimation. In: Proceedings of the 32nd International Conference on Machine Learning, 2015. 125–134

  38. Ramaswamy H G, Scott C, Tewari A. Mixture proportion estimation via kernel embeddings of distributions. In: Proceedings of the 33nd International Conference on Machine Learning, 2016. 2052–2060

  39. du Plessis M C, Niu G, Sugiyama M. Class-prior estimation for learning from positive and unlabeled data. Mach Learn, 2017, 106: 463–492

    Article  MathSciNet  MATH  Google Scholar 

  40. Sriperumbudur B K, Fukumizu K, Gretton A, et al. On the empirical estimation of integral probability metrics. Electron J Statist, 2012, 6: 1550–1599

    Article  MathSciNet  MATH  Google Scholar 

  41. You K C, Wang X M, Long M S, et al. Towards accurate model selection in deep unsupervised domain adaptation. In: Proceedings of International Conference on Machine Learning, 2019. 7124–7133

  42. Jamieson K G, Talwalkar A. Non-stochastic best arm identification and hyperparameter optimization. In: Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, 2016. 240–248

  43. Li L S, Jamieson K G, DeSalvo G, et al. Hyperband: a novel bandit-based approach to hyperparameter optimization. J Mach Learn Res, 2017, 18: 1–52

    MathSciNet  MATH  Google Scholar 

  44. Ke G L, Meng Q, Finley T, et al. LightGBM: a highly efficient gradient boosting decision tree. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017. 3149–3157

  45. Komer B, Bergstra J, Eliasmith C. Hyperopt-sklearn: automatic hyperparameter configuration for scikit-learn. In: Proceedings of ICML workshop on AutoML, 2014

  46. Chang C C, Lin C J. LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol, 2011, 2: 1–27

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant No. 61772262). The authors want to thank the associate editor and reviewers for their helpful comments and suggestions.

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, Nanjing University, Nanjing, 210023, China

    Tong Wei & Yufeng Li

  2. 4Paradigm Inc., Beijing, 100089, China

    Hai Wang & Weiwei Tu

Authors
  1. Tong Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weiwei Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yufeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yufeng Li.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wei, T., Wang, H., Tu, W. et al. Robust model selection for positive and unlabeled learning with constraints. Sci. China Inf. Sci. 65, 212101 (2022). https://doi.org/10.1007/s11432-020-3167-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-020-3167-1

Keywords

Search

Navigation