Density Estimators for Positive-Unlabeled Learning

  • Teresa M. A. Basile
  • Nicola Di Mauro
  • Floriana Esposito
  • Stefano Ferilli
  • Antonio Vergari
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10785)


Positive-Unlabeled (PU) learning works by considering a set of positive samples, and a (usually larger) set of unlabeled ones. This challenging setting requires algorithms to cleverly exploit dependencies hidden in the unlabeled data in order to build models able to accurately discriminate between positive and negative samples. We propose to exploit probabilistic generative models to characterize the distribution of the positive samples, and to label as reliable negative samples those that are in the lowest density regions with respect to the positive ones. The overall framework is flexible enough to be applied to many domains by leveraging tools provided by years of research from the probabilistic generative model community. Results on several benchmark datasets show the performance and flexibility of the proposed approach.


  1. 1.
    Balasubramanian, V.: MDL, Bayesian inference, and the geometry of the space of probability distributions. In: Grünwald, P.D., Myung, I.J., Pitt, M.A. (eds.) Advances in Minimum Description Length: Theory and Applications, pp. 81–98. MIT Press, Cambridge (2005)Google Scholar
  2. 2.
    Basile, T., Mauro, N.D., Esposito, F., Ferilli, S., Vergari, A.: Generative probabilistic models for positive-unlabeled learning. In: Workshop on NFMCP Held with ECML/PKDD (2017)Google Scholar
  3. 3.
    Bengio, Y., Courville, A.C., Vincent, P.: Unsupervised feature learning and deep learning: a review and new perspectives. CoRR abs/1206.5538 (2012)Google Scholar
  4. 4.
    Calvo, B., Larraaga, P., Lozano, J.A.: Learning bayesian classifiers from positive and unlabeled examples. Pattern Recogn. Lett. 28(16), 2375–2384 (2007)CrossRefGoogle Scholar
  5. 5.
    Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. 41(3), 15:1–15:58 (2009)CrossRefGoogle Scholar
  6. 6.
    Chow, C., Liu, C.: Approximating discrete probability distributions with dependence trees. IEEE Trans. Inf. Theor. 14, 462–467 (1968)CrossRefzbMATHGoogle Scholar
  7. 7.
    Cooper, G.F., Herskovits, E.: A Bayesian method for the induction of probabilistic networks from data. Mach. Learn. 9(4), 309–347 (1992)zbMATHGoogle Scholar
  8. 8.
    De Comité, F., Denis, F., Gilleron, R., Letouzey, F.: Positive and unlabeled examples help learning. In: Watanabe, O., Yokomori, T. (eds.) ALT 1999. LNCS (LNAI), vol. 1720, pp. 219–230. Springer, Heidelberg (1999). CrossRefGoogle Scholar
  9. 9.
    Di Mauro, N., Vergari, A., Basile, T.M.A., Esposito, F.: Fast and accurate density estimation with extremely randomized cutset networks. In: Ceci, M., Hollmén, J., Todorovski, L., Vens, C., Džeroski, S. (eds.) ECML PKDD 2017. LNCS (LNAI), vol. 10534, pp. 203–219. Springer, Cham (2017). CrossRefGoogle Scholar
  10. 10.
    Di Mauro, N., Vergari, A., Basile, T.M.A.: Learning Bayesian random cutset forests. In: Esposito, F., Pivert, O., Hacid, M.-S., Raś, Z.W., Ferilli, S. (eds.) ISMIS 2015. LNCS (LNAI), vol. 9384, pp. 122–132. Springer, Cham (2015). CrossRefGoogle Scholar
  11. 11.
    Di Mauro, N., Vergari, A., Esposito, F.: Learning accurate cutset networks by exploiting decomposability. In: Gavanelli, M., Lamma, E., Riguzzi, F. (eds.) AI*IA 2015. LNCS (LNAI), vol. 9336, pp. 221–232. Springer, Cham (2015). CrossRefGoogle Scholar
  12. 12.
    Elkan, C., Noto, K.: Learning classifiers from only positive and unlabeled data. In: KDD, pp. 213–220 (2008)Google Scholar
  13. 13.
    Friedman, N., Geiger, D., Goldszmidt, M.: Bayesian network classifiers. Mach. Learn. 29(2–3), 131–163 (1997)CrossRefzbMATHGoogle Scholar
  14. 14.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer, New York (2009). CrossRefzbMATHGoogle Scholar
  15. 15.
    Hempstalk, K., Frank, E., Witten, I.H.: One-class classification by combining density and class probability estimation. In: Daelemans, W., Goethals, B., Morik, K. (eds.) ECML PKDD 2008. LNCS (LNAI), vol. 5211, pp. 505–519. Springer, Heidelberg (2008). CrossRefGoogle Scholar
  16. 16.
    Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Hoi, C.H., Chan, C.H., Huang, K., Lyu, M.R., King, I.: Biased support vector machine for relevance feedback in image retrieval. In: IJCNN, pp. 3189–3194 (2004)Google Scholar
  18. 18.
    Ienco, D., Pensa, R.G.: Positive and unlabeled learning in categorical data. Neurocomputing 196, 113–124 (2016)CrossRefGoogle Scholar
  19. 19.
    Ienco, D., Pensa, R.G., Meo, R.: From context to distance: learning dissimilarity for categorical data clustering. TKDD 6(1), 1:1–1:25 (2012)CrossRefGoogle Scholar
  20. 20.
    Koller, D., Friedman, N.: Probabilistic Graphical Models: Principles and Techniques. MIT Press, Cambridge (2009)zbMATHGoogle Scholar
  21. 21.
    Li, H., Chen, Z., Liu, B., Wei, X., Shao, J.: Spotting fake reviews via collective positive-unlabeled learning. In: ICDM, pp. 899–904 (2014)Google Scholar
  22. 22.
    Liu, B., Dai, Y., Li, X., Lee, W.S., Yu, P.S.: Building text classifiers using positive and unlabeled examples. In: ICDM, pp. 179–188 (2003)Google Scholar
  23. 23.
    Liu, B., Lee, W.S., Yu, P.S., Li, X.: Partially supervised classification of text documents. In: ICML, pp. 387–394 (2002)Google Scholar
  24. 24.
    Lowd, D., Rooshenas, A.: The Libra toolkit for probabilistic models. CoRR abs/1504.00110 (2015)Google Scholar
  25. 25.
    Meila, M., Jordan, M.I.: Learning with mixtures of trees. JMLR 1, 1–48 (2000)MathSciNetzbMATHGoogle Scholar
  26. 26.
    du Plessis, M.C., Sugiyama, M.: Semi-supervised learning of class balance under class-prior change by distribution matching. Neural Netw. 50, 110–119 (2014)CrossRefzbMATHGoogle Scholar
  27. 27.
    Riahi, F., Schulte, O., Li, Q.: A proposal for statistical outlier detection in relational structures. In: SRAI AAAI Workshop (2014)Google Scholar
  28. 28.
    Schölkopf, B., Platt, J.C., Shawe-Taylor, J.C., Smola, A.J., Williamson, R.C.: Estimating the support of a high-dimensional distribution. Neural Comput. 13(7), 1443–1471 (2001)CrossRefzbMATHGoogle Scholar
  29. 29.
    Tax, D.M.J., Duin, R.P.W.: Support vector data description. Mach. Learn. 54(1), 45–66 (2004)CrossRefzbMATHGoogle Scholar
  30. 30.
    Vergari, A., Di Mauro, N., Esposito, F.: Visualizing and understanding sum-product networks. CoRR abs/1608.08266 (2016)Google Scholar
  31. 31.
    Vergari, A., Di Mauro, N., Esposito, F.: Simplifying, regularizing and strengthening sum-product network structure learning. In: Appice, A., Rodrigues, P.P., Santos Costa, V., Gama, J., Jorge, A., Soares, C. (eds.) ECML PKDD 2015. LNCS (LNAI), vol. 9285, pp. 343–358. Springer, Cham (2015). CrossRefGoogle Scholar
  32. 32.
    Xu, J., Shelton, C.R.: Intrusion detection using continuous time Bayesian networks. J. Artif. Int. Res. 39(1), 745–774 (2010)MathSciNetzbMATHGoogle Scholar
  33. 33.
    Yang, E., Baker, Y., Ravikumar, P., Allen, G., Liu, Z.: Mixed graphical models via exponential families. In: AISTATS, pp. 1042–1050 (2014)Google Scholar
  34. 34.
    Yang, P., Li, X.L., Mei, J.P., Kwoh, C.K., Ng, S.K.: Positive-unlabeled learning for disease gene identification. Bioinformatics 28, 2640–2647 (2012)CrossRefGoogle Scholar
  35. 35.
    Zhao, Y., Kong, X., Philip, S.Y.: Positive and unlabeled learning for graph classification. In: ICDM, pp. 962–971 (2011)Google Scholar
  36. 36.
    Zhou, J., Pan, S., Mao, Q., Tsang, I.: Multi-view positive and unlabeled learning. In: ACML, pp. 555–570 (2012)Google Scholar
  37. 37.
    Zhou, K., Gui-Rong, X., Yang, Q., Yu, Y.: Learning with positive and unlabeled examples using topic-sensitive PLSA. TKDE 22(1), 46–58 (2010)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of Bari “Aldo Moro”BariItaly
  2. 2.Department of PhysicsUniversity of Bari “Aldo Moro”BariItaly
  3. 3.National Institute for Nuclear Physics (INFN)BariItaly

Personalised recommendations