Skip to main content
SpringerLink
Log in

Estimation of mutual information by the fuzzy histogram

  • Published:
Fuzzy Optimization and Decision Making Aims and scope Submit manuscript

Abstract

Mutual Information (MI) is an important dependency measure between random variables, due to its tight connection with information theory. It has numerous applications, both in theory and practice. However, when employed in practice, it is often necessary to estimate the MI from available data. There are several methods to approximate the MI, but arguably one of the simplest and most widespread techniques is the histogram-based approach. This paper suggests the use of fuzzy partitioning for the histogram-based MI estimation. It uses a general form of fuzzy membership functions, which includes the class of crisp membership functions as a special case. It is accordingly shown that the average absolute error of the fuzzy-histogram method is less than that of the naïve histogram method. Moreover, the accuracy of our technique is comparable, and in some cases superior to the accuracy of the Kernel density estimation (KDE) method, which is one of the best MI estimation methods. Furthermore, the computational cost of our technique is significantly less than that of the KDE. The new estimation method is investigated from different aspects, such as average error, bias and variance. Moreover, we explore the usefulness of the fuzzy-histogram MI estimator in a real-world bioinformatics application. Our experiments show that, in contrast to the naïve histogram MI estimator, the fuzzy-histogram MI estimator is able to reveal all dependencies between the gene-expression data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  • Ang, K. K., Chin, Z. Y., Zhang, H., & Guan, C. (2012). Mutual information-based selection of optimal spatial-temporal patterns for single-trial eeg-based bcis. Pattern Recognition, 45(6), 2137–2144.

    Article  MATH  Google Scholar 

  • Crouzet, J. F., & Strauss, O. (2011). Interval-valued probability density estimation based on quasi-continuous histograms: Proof of the conjecture. Fuzzy Sets and Systems, 183(1), 92–100.

    Article  MATH  MathSciNet  Google Scholar 

  • Darbellay, G. (2000). Entropy expressions for multivariate continuous distributions. IEEE Transactions on Information Theory, 46(2), 709–712.

    Article  MATH  MathSciNet  Google Scholar 

  • Darbellay, G. A., & Vajda, I. (1999). Estimation of the information by an adaptive partitioning of the observation space. IEEE Transactions on Information Theory, 45(4), 1315–1321.

    Article  MATH  MathSciNet  Google Scholar 

  • Hughes, T. R. (2012). Supplementary data file of gene expression. http://hugheslab.ccbr.utoronto.ca/supplementary-data/rii/. [Online; Accessed 20 Dec 2012].

  • Hughes, T. R., Marton, M. J., Jones, A. R., Roberts, C. J., Stoughton, R., Armour, C. D., et al. (2000). Functional discovery via a compendium of expression profiles. Cell, 102(1), 109–126.

    Article  Google Scholar 

  • Karasuyama, M., & Sugiyama, M. (2012). Canonical dependency analysis based on squared-loss mutual information. Neural Networking, 34, 46–55.

    Google Scholar 

  • Kraskov, A., Stögbauer, H., & Grassberger, P. (2004). Estimating mutual information. Physical Review E, 69(6), 066138.

    Article  MathSciNet  Google Scholar 

  • Loquin, K., & Strauss, O. (2006). Fuzzy histograms and density estimation. In J. Lawry, E. Miranda, A. Bugarin, S. Li, M. A. Gil, P. Grzegorzewski, & O. Hyrniewicz (Eds.), Soft methods for integrated uncertainty modelling, volume 37 of advances in soft computing (pp. 45–52). Berlin Heidelberg: Springer.

    Chapter  Google Scholar 

  • Loquin, K., & Strauss, O. (2008). Histogram density estimators based upon a fuzzy partition. Statistics and Probability Letters, 78(13), 1863–1868.

    Google Scholar 

  • Moddemeijer, R. (1989). On estimation of entropy and mutual information of continuous distributions. Signal Processing, 16(3), 233–248.

    Article  MathSciNet  Google Scholar 

  • Moon, Y. I., Rajagopalan, B., & Lall, U. (1995). Estimation of mutual information using kernel density estimators. Physical Review E, 52(3), 2318–2321.

    Article  Google Scholar 

  • Schaffernicht, E., Kaltenhaeuser, R., Verma, S., & Gross, H. M. (2010). On estimating mutual information for feature selection. Artificial Neural Networks-ICANN, 2010, 362–367.

    Google Scholar 

  • Steuer, R., Kurths, J., Daub, C. O., Weise, J., & Selbig, J. (2002). The mutual information: Detecting and evaluating dependencies between variables. Bioinformatics, 18(suppl 2), S231–S240.

    Article  Google Scholar 

  • Tenekedjiev, K., & Nikolova, N. (2008). Justification and numerical realization of the uniform method for finding point estimates of interval elicited scaling constants. Fuzzy Optimization and Decision Making, 7(2), 119–145.

    Article  MATH  MathSciNet  Google Scholar 

  • Wang, Q., Shen, Y., & Zhang, J. Q. (2005). A nonlinear correlation measure for multivariable data set. Physica D: Nonlinear Phenomena, 200(3–4), 287–295.

    Google Scholar 

  • Zografos, K., & Nadarajah, S. (2005). Expressions for rényi and shannon entropies for multivariate distributions. Statistics and Probability Letters, 71(1), 71–84.

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering and Information Technology, Amirkabir University of Technology, Tehran, Iran

    Maryam Amir Haeri & Mohammad Mehdi Ebadzadeh

Authors
  1. Maryam Amir Haeri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Mehdi Ebadzadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammad Mehdi Ebadzadeh.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Amir Haeri, M., Ebadzadeh, M.M. Estimation of mutual information by the fuzzy histogram. Fuzzy Optim Decis Making 13, 287–318 (2014). https://doi.org/10.1007/s10700-014-9178-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10700-014-9178-0

Keywords

Search

Navigation