Estimation of Probability Density Function, Differential Entropy and Other Relative Quantities for Data Streams with Concept Drift

  • Maciej JaworskiEmail author
  • Patryk Najgebauer
  • Piotr Goetzen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10842)


In this paper estimators of nonstationary probability density function are proposed. Additionally, applying the trapezoidal method of numerical integration, the estimators of two information-theoretic measures are presented: the differential entropy and the Renyi’s quadratic differential entropy. Finally, using an analogous methodology, estimators of the Cauchy-Schwarz divergence and the probability density function divergence are proposed, which are used to measure the differences between two probability density functions. All estimators are proposed in two variants: one with the sliding window and one with the forgetting factor. Performance of all the estimators is verified using numerical simulations.


Data stream Concept drift Density estimation Differential entropy Kernel function 



This work was supported by the Polish National Science Centre under Grant No. 2014/15/B/ST7/05264.


  1. 1.
    Bilski, J., Smolag, J.: Parallel architectures for learning the RTRN and Elman dynamic neural networks. IEEE Trans. Parallel Distrib. Syst. 26(9), 2561–2570 (2015)CrossRefGoogle Scholar
  2. 2.
    Chang, O., Constante, P., Gordon, A., Singana, M.: A novel deep neural network that uses space-time features for tracking and recognizing a moving object. J. Artif. Intell. Soft Comput. Res. 7(2), 125–136 (2017)CrossRefGoogle Scholar
  3. 3.
    Devi, V.S., Meena, L.: Parallel MCNN (PMCNN) with application to prototype selection on large and streaming data. J. Artif. Intell. Soft Comput. Res. 7(3), 155–169 (2017)CrossRefGoogle Scholar
  4. 4.
    Devroye, L.P.: On the pointwise and the integral convergence of recursive kernel estimates of probability densities. Utilitas Math. (Canada) 15, 113–128 (1979)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Ditzler, G., Roveri, M., Alippi, C., Polikar, R.: Learning in nonstationary environments: a survey. IEEE Comput. Intell. Mag. 10(4), 12–25 (2015)CrossRefGoogle Scholar
  6. 6.
    Duda, P., Jaworski, M., Rutkowski, L.: On ensemble components selection in data streams scenario with reoccurring concept-drift. In: 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1821–1827, November 2017Google Scholar
  7. 7.
    Epanechnikov, V.A.: Non-parametric estimation of a multivariate probability density. Theory Probab. Appl. 14(1), 153–158 (1969)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Galkowski, T., Rutkowski, L.: Nonparametric fitting of multivariate functions. IEEE Trans. Autom. Control 31(8), 785–787 (1986)CrossRefGoogle Scholar
  9. 9.
    Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., Bouchachia, A.: A survey on concept drift adaptation. ACM Comput. Surv. (CSUR) 46(4), 44 (2014)CrossRefGoogle Scholar
  10. 10.
    Greblicki, W., Pawlak, M.: Nonparametric System Identification. Cambridge University Press, Cambridge (2008)CrossRefGoogle Scholar
  11. 11.
    Jaworski, M., Duda, P., Rutkowski, L.: On applying the Restricted Boltzmann Machine to active concept drift detection. In: 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 3512–3519, November 2017Google Scholar
  12. 12.
    Jaworski, M., Duda, P., Rutkowski, L.: New splitting criteria for decision trees in stationary data streams. IEEE Trans. Neural Netw. Learn. Syst. PP(99), 1–14 (2018)MathSciNetGoogle Scholar
  13. 13.
    Jaworski, M., Duda, P., Rutkowski, L., Najgebauer, P., Pawlak, M.: Heuristic regression function estimation methods for data streams with concept drift. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2017. LNCS (LNAI), vol. 10246, pp. 726–737. Springer, Cham (2017). Scholar
  14. 14.
    Krzyzak, A., Pawlak, M.: The pointwise rate of convergence of the kernel regression estimate. J. Stat. Plan. Inference 16, 159–166 (1987)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Lemaire, V., Salperwyck, C., Bondu, A.: A survey on supervised classification on data streams. In: Zimányi, E., Kutsche, R.-D. (eds.) eBISS 2014. LNBIP, vol. 205, pp. 88–125. Springer, Cham (2015). Scholar
  16. 16.
    Napoli, C., Pappalardo, G., Tramontana, E., Nowicki, R.K., Starczewski, J.T., Woźniak, M.: Toward work groups classification based on probabilistic neural network approach. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2015. LNCS (LNAI), vol. 9119, pp. 79–89. Springer, Cham (2015). Scholar
  17. 17.
    Notomista, G., Botsch, M.: A machine learning approach for the segmentation of driving maneuvers and its application in autonomous parking. J. Artif. Intell. Soft Comput. Res. 7(4), 243–255 (2017)CrossRefGoogle Scholar
  18. 18.
    Parzen, E.: On estimation of probability density function and mode. Ann. Math. Stat. 33, 1065–1076 (1962)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Pietruczuk, L., Rutkowski, L., Jaworski, M., Duda, P.: The Parzen kernel approach to learning in non-stationary environment. In: 2014 International Joint Conference on Neural Networks (IJCNN), pp. 3319–3323 (2014)Google Scholar
  20. 20.
    Rutkowski, L.: Sequential estimates of a regression function by orthogonal series with applications in discrimination. In: Révész, P., Schatterer, L., Zolotarev, V.M. (eds.) The First Pannonian Symposium on Mathematical Statistics. LNS, vol. 8, pp. 236–244. Springer, New York (1981). Scholar
  21. 21.
    Rutkowski, L.: Generalized regression neural networks in time-varying environment. IEEE Trans. Neural Netw. 15, 576–596 (2004)CrossRefGoogle Scholar
  22. 22.
    Rutkowski, L., Jaworski, M., Pietruczuk, L., Duda, P.: A new method for data stream mining based on the misclassification error. IEEE Trans. Neural Netw. Learn. Syst. 26(5), 1048–1059 (2015)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Specht, D.F.: Probabilistic neural networks. Neural Netw. 3(1), 109–118 (1990)CrossRefGoogle Scholar
  24. 24.
    Yan, P.: Mapreduce and semantics enabled event detection using social media. J. Artif. Intell. Soft Comput. Res. 7(3), 201–213 (2017)CrossRefGoogle Scholar
  25. 25.
    Yang, S., Sato, Y.: Swarm intelligence algorithm based on competitive predators with dynamic virtual teams. J. Artif. Intell. Soft Comput. Res. 7(2), 87–101 (2017)CrossRefGoogle Scholar
  26. 26.
    Zalasiński, M., Cpałka, K.: New algorithm for on-line signature verification using characteristic hybrid partitions. In: Wilimowska, Z., Borzemski, L., Grzech, A., Świątek, J. (eds.) ISAT 2015. AISC, vol. 432, pp. 147–157. Springer, Cham (2016). Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Maciej Jaworski
    • 1
    Email author
  • Patryk Najgebauer
    • 1
  • Piotr Goetzen
    • 2
    • 3
  1. 1.Institute of Computational IntelligenceCzestochowa University of TechnologyCzestochowaPoland
  2. 2.Information Technology InstituteUniversity of Social SciencesŁódźPoland
  3. 3.Clark UniversityWorcesterUSA

Personalised recommendations