×

Statistical estimation of mutual information for mixed model. (English) Zbl 1477.62074

Summary: Asymptotic unbiasedness and \(L^2\)-consistency are established for various statistical estimates of mutual information in the mixed models framework. Such models are important, e.g., for analysis of medical and biological data. The study of the conditional Shannon entropy as well as new results devoted to statistical estimation of the differential Shannon entropy are employed essentially. Theoretical results are completed by computer simulations for logistic regression model with different parameters. The numerical experiments demonstrate that new statistics, proposed by the authors, have certain advantages.

MSC:

62G05 Nonparametric estimation
62G20 Asymptotic properties of nonparametric inference
62J05 Linear regression; mixed models
60F25 \(L^p\)-limit theorems
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Balagani, K.; Phoha, V., On the feature selection criterion based on an approximation of multidimensional mutual information, IEEE Trans Pattern Anal Mach Intell, 32, 7, 1342-1343 (2010) · doi:10.1109/TPAMI.2010.62
[2] Bennasar, M.; Hicks, Y.; Setchi, R., Feature selection using joint mutual information maximisation, Expert Syst Appl, 42, 8520-8532 (2014) · doi:10.1016/j.eswa.2015.07.007
[3] Berrett TB, Samworth RJ, Yuan M (2016) Efficient multivariate entropy estimation via k-nearest neighbour distances. arXiv:160600304 · Zbl 1473.62177
[4] Biau, G.; Devroy, L., Lectures on the nearest neighbor method (2015), New York: Springer-Verlag, New York · Zbl 1330.68001 · doi:10.1007/978-3-319-25388-6
[5] Bulinski A, Dimitrov D (2019a) Statistical estimation of the Kullback - Leibler divergence. arXiv:190700196 · Zbl 1411.60049
[6] Bulinski A, Dimitrov D (2019b) Statistical estimation of the Shannon entropy. Acta Math Sinica English Series 35:17-46 · Zbl 1411.60049
[7] Bulinski, A.; Kozhevin, A., Modification of the MDR-EFE method for stratified samples, Stat, Optim Inf Comput, 5, 1-18 (2017) · doi:10.19139/soic.v5i1.277
[8] Bulinski A, Kozhevin A (2018) Statistical estimation of conditional Shannon entropy. ESAIM: Probability and Statistics pp 1-35 , published online: November 28. https://doi.org/10.1051.ps.2018026 · Zbl 1418.60026
[9] Coelho, F.; Braga, A.; Verleysen, M., A mutual information estimator for continuous and discrete variables applied to feature selection and classification problems, Int J Computat Int Syst, 9, 4, 726-733 (2016) · doi:10.1080/18756891.2016.1204120
[10] Delattre S, Fournier N (2017) On the Kozachenko-Leonenko entropy estimator. Journal of Statistical Planning and Inference 185 · Zbl 1359.62098
[11] Doquire G, Verleysen M (2012) A comparison of mutual information estimators for feature selection. In: Proc. of the 1st international conference on pattern recognition applications and methods, pp 176-185. doi:10.5220/0003726101760185
[12] Favatti, P.; Lotti, G.; Romani, F., Algorithm 691: Improving quadpack automatic integration routines, ACM Trans Math Softw, 17, 2, 218-232 (1991) · Zbl 0900.65050 · doi:10.1145/108556.108580
[13] Fleuret, F., Fast binary feature selection with conditional mutual information, J Mach Learn Res, 4, 1531-1555 (2004) · Zbl 1222.68200
[14] Gao W, Kannan S, Oh S, Viswanath P (2017) Estimating mutual information for discrete-continuous mixtures. In: 31st conference on neural information processing systems (NIPS), Long Beach, CA, USA, pp 1-12
[15] Hoeffding, W., Probability inequalities for sums of bounded random variables, J Amer Statist Assoc, 58, 301, 13-30 (1963) · Zbl 0127.10602 · doi:10.1080/01621459.1963.10500830
[16] Kozachenko, L.; Leonenko, N., Sample estimate of the entropy of a random vector, Probl Inf Transm, 23, 95-101 (1987) · Zbl 0633.62005
[17] Kraskov, A.; Stögbauer, H.; Grassberger, P., Estimating mutual information, Phys Rev E, 69, 066,138 (2004) · doi:10.1103/PhysRevE.69.066138
[18] Macedo, F.; Oliveira, R.; Pacheco, A.; Valadas, R., Theoretical foundations of forward feature selection methods based on mutual information, Neurocomputing, 325, 67-89 (2019) · doi:10.1016/j.neucom.2018.09.077
[19] Massaron, L.; Boschetti, A., Regression analysis with python (2016), Birmingham: Packt Publishing Ltd., Birmingham
[20] Nair C, Prabhakar B, Shah D (2007) Entropy for mixtures of discrete and continuous variables. arXiv:0607075v2
[21] Novovičová J, Somol P, Haind M, Pudin P (2007) Conditional mutual information based feature selection for classification task. In: Ruedz M D L, Kittler J (eds) CIARP 2007, LNCS, vol 4756. Springer-Verlag, Berlin, pp 417-426
[22] Paninski, L., Estimation of entropy and mutual information, Neural Comput., 15, 1191-1253 (2003) · Zbl 1052.62003 · doi:10.1162/089976603321780272
[23] Peng, H.; Long, F.; Ding, C., Feature selection based on mutual information criteria of maxdependency, max-relevance, and min-redundancy, IEEE Trans. Pattern Anal. Mach. Intell., 27, 1226-1238 (2005) · doi:10.1109/TPAMI.2005.159
[24] Vergara, J.; Estévez, P., A review of feature selection methods based on mutual information, Neural Comput Appl, 24, 175-186 (2014) · doi:10.1007/s00521-013-1368-0
[25] Verleysen M, Rossi F, Franëcois D (2009) Advances in feature selection with mutual information. arXiv:09090635v1
[26] Yeh, J., Real analysis: Theory of measure and integration (2014), Singapore: World Scientific, Singapore · Zbl 1301.26002 · doi:10.1142/9037
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.