Data Mining and Knowledge Discovery

, Volume 32, Issue 3, pp 561–603 | Cite as

Online estimation of discrete, continuous, and conditional joint densities using classifier chains

  • Michael GeilkeEmail author
  • Andreas Karwath
  • Eibe Frank
  • Stefan Kramer


We address the problem of estimating discrete, continuous, and conditional joint densities online, i.e., the algorithm is only provided the current example and its current estimate for its update. The family of proposed online density estimators, estimation of densities online (EDO), uses classifier chains to model dependencies among features, where each classifier in the chain estimates the probability of one particular feature. Because a single chain may not provide a reliable estimate, we also consider ensembles of classifier chains and ensembles of weighted classifier chains. For all density estimators, we provide consistency proofs and propose algorithms to perform certain inference tasks. The empirical evaluation of the estimators is conducted in several experiments and on datasets of up to several millions of instances. In the discrete case, we compare our estimators to density estimates computed by Bayesian structure learners. In the continuous case, we compare them to a state-of-the-art online density estimator. Our experiments demonstrate that, even though designed to work online, EDO delivers estimators of competitive accuracy compared to other density estimators (batch Bayesian structure learners on discrete datasets and the state-of-the-art online density estimator on continuous datasets). Besides achieving similar performance in these cases, EDO is also able to estimate densities with mixed types of variables, i.e., discrete and continuous random variables.


Data streams Density estimation Classifier chains Inference 



We would like to thank the editor and the anonymous reviewers for their comments. They improved the presentation, readability, and quality of this paper substantially. We are particularly grateful to the anonymous reviewer who proposed the exponentiated gradient investment strategy for weighting the classifier chains.

Supplementary material

10618_2017_546_MOESM1_ESM.pdf (64 kb)
Supplementary material 1 (pdf 63 KB)
10618_2017_546_MOESM2_ESM.pdf (209 kb)
Supplementary material 2 (pdf 209 KB)
10618_2017_546_MOESM3_ESM.pdf (81 kb)
Supplementary material 3 (pdf 80 KB)


  1. Bauer E, Kohavi R (1999) An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Mach Learn 36(1–2):105–139CrossRefGoogle Scholar
  2. Bifet A, Holmes G, Pfahringer B, Kranen P, Kremer H, Jansen T, Seidl T (2010) MOA: massive online analysis, a framework for stream classification and clustering. J Mach Learn Res Proc Track 11:44–50Google Scholar
  3. Blum A (1996) On-line algorithms in machine learning. In: Proceedings of the workshop on On-line Algorithms, Dagstuhl. Springer, pp 306–325Google Scholar
  4. Buchwald F, Girschick T, Frank E, Kramer S (2010) Fast conditional density estimation for quantitative structure-activity relationships. In: Proceedings of the twenty-fourth AAAI conference on artificial intelligence, pp 1268–1273Google Scholar
  5. Cesa-Bianchi N, Lugosi G (2006) Prediction, learning, and games. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  6. Chakraborty S (2008) Some applications of dirac’s delta function in statistics for more than one random variable. Appl Appl Math Int J (AAM) 3(1):4254MathSciNetzbMATHGoogle Scholar
  7. Cheng MY, Gasser T, Hall P (1999) Nonparametric density estimation under unimodality and monotonicity constraints. J Comput Graph Stat 8(1):1–21MathSciNetGoogle Scholar
  8. Cover TM, Thomas JA (2006) Elements of information theory, 2nd edn. Wiley, New YorkzbMATHGoogle Scholar
  9. Davies S, Moore AW (2002) Interpolating conditional density trees. In: Uncertainty in artificial intelligence, pp 119–127Google Scholar
  10. Dembczynski K, Cheng W, Hüllermeier E (2010) Bayes optimal multilabel classification via probabilistic classifier chains. In: International conference on machine learning, pp 279–286Google Scholar
  11. Dembczynski K, Waegeman W, Hüllermeier E (2012) An analysis of chaining in multi-label classification. In: Proceedings of the 20th European conference on artificial intelligence (ECAI 2012), pp 294–299Google Scholar
  12. Dembczynski K, Kotlowski W, Waegeman W, Busa-Fekete R, Hüllermeier E (2016) Consistency of probabilistic classifier trees. In: Proceedings of the 2016 European conference on machine learning and knowledge discovery in databases (ECML PKDD 2016), pp 511–526Google Scholar
  13. Domingos P, Hulten G (2000) Mining high-speed data streams. In: Knowledge discovery and data mining, pp 71–80Google Scholar
  14. Elgammal A, Duraiswami R, Davis LS (2003) Efficient kernel density estimation using the fast gauss transform with applications to color modeling and tracking. IEEE Trans Pattern Anal Mach Intell 25:1499–1504CrossRefGoogle Scholar
  15. Frank E, Bouckaert RR (2009) Conditional density estimation with class probability estimators. In: Proceedings of first Asian conference on machine learning, pp 65–81Google Scholar
  16. Frank E, Kramer S (2004) Ensembles of nested dichotomies for multi-class problems. In: Proceedings of the 21st international conference of machine learning, pp 305–312Google Scholar
  17. Friedman N, Goldszmidt M (1996) Learning bayesian networks with local structure. In: Proceedings of the twelfth annual conference on uncertainty in artificial intelligence (UAI ’96), pp 252–262Google Scholar
  18. Gama J, Pinto C (2006) Discretization from data streams: applications to histograms and data mining. In: SAC, pp 662–667Google Scholar
  19. Geilke M, Karwath A, Frank E, Kramer S (2013) Online estimation of discrete densities. In: Proceedings of the 13th IEEE international conference on data mining, pp 191–200Google Scholar
  20. Geilke M, Karwath A, Kramer S (2014) A probabilistic condensed representation of data for stream mining. In: Proceedings of the 2014 international conference on data science and advanced analytics (DSAA 2014), IEEE, pp 297–303Google Scholar
  21. Geilke M, Karwath A, Kramer S (2015) Modeling recurrent distributions in streams using possible worlds. In: Proceedings of the 2015 international conference on data science and advanced analytics (DSAA 2015), pp 1–9Google Scholar
  22. Goldberger J, Roweis ST (2004) Hierarchical clustering of a mixture model. Adv Neural Inf Process Syst 17:505–512Google Scholar
  23. Hall P, Presnell B (1999) Density estimation under constraints. J Comput Graph Stat 8(2):259–277MathSciNetGoogle Scholar
  24. Holmes MP, Gray AG, Isbell CL Jr (2012) Fast nonparametric conditional density estimation. CoRR arXiv:abs/1206.5278
  25. Hulten G, Spencer L, Domingos P (2001) Mining time-changing data streams. In: Knowledge discovery and data mining, pp 97–106Google Scholar
  26. Hwang JN, Lay SR, Lippman A (1994) Nonparametric multivariate density estimation: a comparative study. IEEE Trans Signal Process 42(10):2795–2810CrossRefGoogle Scholar
  27. Kim J, Scott CD (2012) Robust kernel density estimation. J Mach Learn Res 13:2529–2565MathSciNetzbMATHGoogle Scholar
  28. Kristan M, Leonardis A (2010) Online discriminative kernel density estimation. In: International conference on pattern recognition, pp 581–584Google Scholar
  29. Kristan M, Leonardis A, Skocaj D (2011) Multivariate online kernel density estimation with gaussian kernels. Pattern Recogn 44(10–11):2630–2642zbMATHCrossRefGoogle Scholar
  30. Kumar A, Vembu S, Menon AK, Elkan C (2013) Beam search algorithms for multilabel learning. Mach Learn 92(1):65–89MathSciNetzbMATHCrossRefGoogle Scholar
  31. Lambert CG, Harrington SE, Harvey CR, Glodjo A (1999) Efficient on-line nonparametric kernel density estimation. Algorithmica 25(1):37–57MathSciNetzbMATHCrossRefGoogle Scholar
  32. Littlestone N (1987) Learning quickly when irrelevant attributes abound: a new linear-threshold algorithm. Mach Learn 2(4):285–318Google Scholar
  33. Liu H, Lafferty JD, Wasserman LA (2007) Sparse nonparametric density estimation in high dimensions using the rodeo. In: Proceedings of the eleventh international conference on artificial intelligence and statistics, pp 283–290Google Scholar
  34. Mann TP (2006) Numerically stable hidden Markov model implementation. HMM Scaling Tutor, pp 1–8.Google Scholar
  35. Melançon G, Philippe F (2004) Generating connected acyclic digraphs uniformly at random. Inf Process Lett 90(4):209–213MathSciNetzbMATHCrossRefGoogle Scholar
  36. Motwani R, Raghavan P (1995) Randomized algorithms. Cambridge University Press, New YorkzbMATHCrossRefGoogle Scholar
  37. Peherstorfer B, Pflüger D, Bungartz H (2014) Density estimation with adaptive sparse grids for large data sets. In: Proceedings of the 2014 SIAM international conference on data mining, pp 443–451Google Scholar
  38. Ram P, Gray AG (2011) Density estimation trees. In: Knowledge discovery and data mining, pp 627–635Google Scholar
  39. Rau MM, Seitz S, Brimioulle F, Frank E, Friedrich O, Gruen D, Hoyle B (2015) Accurate photometric redshift probability density estimation—method comparison and application. Monthly Notices R Astron Soc 452(4):3710–3725CrossRefGoogle Scholar
  40. Raykar VC, Duraiswami R (2006) Fast optimal bandwidth selection for kernel density estimation. In: Proceedings of the sixth SIAM international conference on data mining, pp 524–528Google Scholar
  41. Read J, Pfahringer B, Holmes G, Frank E (2011) Classifier chains for multi-label classification. Mach Learn 85(3):333–359MathSciNetCrossRefGoogle Scholar
  42. Scott DW, Sain SR (2004) Multi-dimensional density estimation. Elsevier, Amsterdam, pp 229–263Google Scholar
  43. Scutari M (2010) Learning Bayesian networks with the bnlearn R package. J Stat Softw 35(3):1–22CrossRefGoogle Scholar
  44. Sheather SJ, Jones MC (1991) A reliable data-based bandwidth selection method for kernel density estimation. J R Stat Soc Ser B (Methodol) 53(3):683–690MathSciNetzbMATHGoogle Scholar
  45. Su J, Zhang H (2006) Full Bayesian network classifiers. In: Proceedings of the twenty-third international conference on machine learning, pp 897–904Google Scholar
  46. Valiant LG (1984) A theory of the learnable. Commun ACM 27(11):1134–1142zbMATHCrossRefGoogle Scholar
  47. Vapnik V, Mukherjee S (1999) Support vector method for multivariate density estimation. In: Neural information processing systems, pp 659–665Google Scholar
  48. Wan R, Wang L (2010) Clustering over evolving data stream with mixed attributes. J Comput Inf Syst 6:1555–1562Google Scholar
  49. Wang X, Wang Y (2015) Nonparametric multivariate density estimation using mixtures. Stat Comput 25(2):349–364MathSciNetzbMATHCrossRefGoogle Scholar
  50. Wied D, Weißbach R (2012) Consistency of the kernel density estimator: a survey. Stat Papers 53(1):1–21MathSciNetzbMATHCrossRefGoogle Scholar
  51. Wu K, Zhang K, Fan W, Edwards A, Yu PS (2014) RS-forest: a rapid density estimator for streaming anomaly detection. In: Proceedings of the 14th international conference on data mining, pp 600–609Google Scholar
  52. Zhou A, Cai Z, Wei L, Qian W (2003) M-kernel merging: towards density estimation over data streams. In: Proceedings of the eighth international conference on database systems for advanced applications, IEEE computer society, pp 285–292Google Scholar
  53. Zliobaite I, Bifet A, Read J, Pfahringer B, Holmes G (2015) Evaluation methods and decision theory for classification of streaming data with temporal dependence. Mach Learn 98(3):455–482MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© The Author(s) 2017

Authors and Affiliations

  • Michael Geilke
    • 1
    Email author
  • Andreas Karwath
    • 1
  • Eibe Frank
    • 2
  • Stefan Kramer
    • 1
  1. 1.Johannes Gutenberg-Universität MainzMainzGermany
  2. 2.Department of Computer ScienceThe University of WaikatoHamiltonNew Zealand

Personalised recommendations