×

Eigenvalues and constraints in mixture modeling: geometric and computational issues. (English) Zbl 1414.62071

Summary: This paper presents a review about the usage of eigenvalues restrictions for constrained parameter estimation in mixtures of elliptical distributions according to the likelihood approach. The restrictions serve a twofold purpose: to avoid convergence to degenerate solutions and to reduce the onset of non interesting (spurious) local maximizers, related to complex likelihood surfaces. The paper shows how the constraints may play a key role in the theory of Euclidean data clustering. The aim here is to provide a reasoned survey of the constraints and their applications, considering the contributions of many authors and spanning the literature of the last 30 years.

MSC:

62F10 Point estimation
62F12 Asymptotic properties of parametric estimators
62F30 Parametric inference under constraints
62F35 Robustness and adaptive procedures (parametric inference)

Software:

mclust; TCLUST
PDFBibTeX XMLCite
Full Text: DOI Link

References:

[1] Andrews JL, McNicholas PD (2011) Extending mixtures of multivariate \[t\] t-factor analyzers. Stat Comput 21:361-373 · Zbl 1255.62171 · doi:10.1007/s11222-010-9175-2
[2] Andrews JL, McNicholas PD, Subedi S (2011) Model-based classification via mixtures of multivariate \[t\] t-distributions. Comput Stat Data Anal 55:520-529 · Zbl 1247.62151 · doi:10.1016/j.csda.2010.05.019
[3] Banfield JD, Raftery AE (1993) Model-based Gaussian and non-Gaussian clustering. Biometrics 49:803-821 · Zbl 0794.62034 · doi:10.2307/2532201
[4] Baudry J-P, Celeux G (2015) EM for mixtures. Stat Comput 22(5):1021-1029 · Zbl 1331.62301
[5] Biernacki C (2004) Initializing EM using the properties of its trajectories in gaussian mixtures. Stat Comput 14:267-279 · doi:10.1023/B:STCO.0000035306.77434.31
[6] Biernacki C, Chrétien S (2003) Degeneracy in the maximum likelihood estimation of univariate Gaussian mixtures with the EM. Stat Probab Lett 61:373-382 · Zbl 1038.62023 · doi:10.1016/S0167-7152(02)00396-6
[7] Biernacki C, Celeux G, Govaert G (2003) Choosing starting values for the em algorithm for getting the highest likelihood in multivariate Gaussian mixture models. Comput Stat Data Anal 41(3-4):561-575 · Zbl 1429.62235 · doi:10.1016/S0167-9473(02)00163-9
[8] Boyles RA (1983) On the convergence of the EM algorithm. J R Stat Soc B 45:47-50 · Zbl 0508.62030
[9] Celeux G, Govaert G (1995) Gaussian parsimonious clustering models. Pattern Recognit 28:781-793 · doi:10.1016/0031-3203(94)00125-6
[10] Cerioli A, García-Escudero LA, Mayo-Iscar A, Riani M (2016) Finding the number of groups in model-based clustering via constrained likelihoods. http://uvadoc.uva.es/handle/10324/18093 · Zbl 07498957
[11] Chanda KC (1954) A note on the consistency and maxima of the roots of likelihood equations. Biometrika 41:56-61 · Zbl 0055.12901 · doi:10.1093/biomet/41.1-2.56
[12] Ciuperca G, Ridolfi A, Idier J (2003) Penalized maximum likelihood estimator for normal mixtures. Scand J Stat 30:45-59 · Zbl 1034.62018 · doi:10.1111/1467-9469.00317
[13] Cramér H (1946) Math Methods Stat. Princeton University Press, Princeton
[14] Day N (1969) Estimating the components of a mixture of normal distributions. Biometrika 56(3):463-474 · Zbl 0183.48106 · doi:10.1093/biomet/56.3.463
[15] Dempster A, Laird N, Rubin D (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Ser B (Methodol) 39(1):1-38 · Zbl 0364.62022
[16] Dennis JE (1981) Algorithms for non linear fitting. In: Proceedings of the NATO advanced research symposium, Cambridge, England. Cambridge University
[17] Dykstra RL (1983) An algorithm for restricted least squares regression. J Am Stat Assoc 78(384):837-842 · Zbl 0535.62063 · doi:10.1080/01621459.1983.10477029
[18] Fang K, Anderson T (1990) Statistical inference in elliptically contoured and related distributions. Alberton, New York · Zbl 0747.00016
[19] Fraley C, Raftery AE (2006) Mclust version 3: an R package for normal mixture modeling and model-based clustering. Technical report, DTIC Document
[20] Fraley C, Raftery A (2007) Bayesian regularization for normal mixture estimation and model-based clustering. J Classif 24(2):155-181 · Zbl 1159.62302 · doi:10.1007/s00357-007-0004-5
[21] Fraley C, Raftery A, Murphy T, Scrucca L (2012) mclust version 4 for R: normal mixture modeling for model-based clustering, classification, and density estimation. University of Washington, Seattle
[22] Fritz H, García-Escudero LA, Mayo-Iscar A (2012) tclust: an R package for a trimming approach to cluster analysis. J Stat Softw 47(12):1-26 · doi:10.18637/jss.v047.i12
[23] Fritz H, García-Escudero LA, Mayo-Iscar A (2013) A fast algorithm for robust constrained clustering. Comput Stat Data Anal 61:124-136 · Zbl 1349.62264 · doi:10.1016/j.csda.2012.11.018
[24] Gallegos MT (2002) Maximum likelihood clustering with outliers. In: Jajuga K, Sokolowski A, Bock H-H (eds) Classification, clustering, and data analysis: recent advances and applications. Springer, Berlin, pp 247-255
[25] Gallegos M, Ritter G (2009) Trimmed ML estimation of contaminated mixture. Sankhya (Ser A) 71:164-220 · Zbl 1193.62021
[26] García-Escudero LA, Gordaliza A, Matrán C, Mayo-Iscar A (2008) A general trimming approach to robust cluster analysis. Ann Stat 36(3):1324-1345 · Zbl 1360.62328 · doi:10.1214/07-AOS515
[27] García-Escudero LA, Gordaliza A, Mayo-Iscar A (2014) A constrained robust proposal for mixture modeling avoiding spurious solutions. Adv Data Anal Classif 8(1):27-43 · Zbl 1459.62110 · doi:10.1007/s11634-013-0153-3
[28] García-Escudero LA, Gordaliza A, Matrán C, Mayo-Iscar A (2015) Avoiding spurious local maximizers in mixture modelling. Stat Comput 25:619-633 · Zbl 1331.62100 · doi:10.1007/s11222-014-9455-3
[29] García-Escudero LA, Gordaliza A, Greselin F, Ingrassia S, Mayo-Iscar A (2016) The joint role of trimming and constraints in robust estimation for mixtures of Gaussian factor analyzers. Comput Stat Data Anal 99:131-147 · Zbl 1505.62152 · doi:10.1016/j.csda.2016.01.005
[30] García-Escudero LA, Gordaliza A, Greselin F, Ingrassia S, Mayo-Iscar A (2017) Eigenvalues in robust approaches to mixture modeling: a review. Technical report, in preparation · Zbl 1505.62152
[31] Ghahramani Z, Hinton G (1997) The EM algorithm for factor analyzers. Technical report CRG-TR-96-1, University of Toronto
[32] Greselin F, Ingrassia S (2010) Constrained monotone em algorithms for mixtures of multivariate \[t\] t-distributions. Stat Comput 20:9-22 · doi:10.1007/s11222-008-9112-9
[33] Greselin F, Ingrassia S (2015) Maximum likelihood estimation in constrained parameter spaces for mixtures of factor analyzers. Stat Comput 25(2):215-226 · Zbl 1331.62307 · doi:10.1007/s11222-013-9427-z
[34] Greselin F, Ingrassia S, Punzo A (2011) Assessing the pattern of covariance matrices via an augmentation multiple testing procedure. Stat Methods Appl 20:141-170 · Zbl 1232.62090 · doi:10.1007/s10260-010-0157-5
[35] Hathaway RJ (1985) A constrained formulation of maximum-likelihood estimation for normal mixture distributions. Ann Stat 13(2):795-800 · Zbl 0576.62039 · doi:10.1214/aos/1176349557
[36] Hathaway RJ (1996) A constrained EM algorithm for univariate normal mixtures. J Stat Comput Simul 23:211-230 · doi:10.1080/00949658608810872
[37] Hennig C (2004) Breakdown points for maximum likelihood estimators of location-scale mixtures. Ann Stat 32:1313-1340 · Zbl 1047.62063 · doi:10.1214/009053604000000571
[38] Ingrassia S (1992) A comparison between the simulated annealing and the EM algorithms in normal mixture decompositions. Stat Comput 2:203-211 · doi:10.1007/BF01889680
[39] Ingrassia S (2004) A likelihood-based constrained algorithm for multivariate normal mixture models. Stat Methods Appl 13(4):151-166 · Zbl 1205.62066
[40] Ingrassia S, Rocci R (2007) Constrained monotone EM algorithms for finite mixture of multivariate gaussians. Comput Stat Data Anal 51(11):5339-5351 · Zbl 1445.62116 · doi:10.1016/j.csda.2006.10.011
[41] Ingrassia S, Rocci R (2011) Degeneracy of the EM algorithm for the MLE of multivariate Gaussian mixtures and dynamic constraints. Comput Stat Data Anal 55(4):1715-1725 · Zbl 1328.65030 · doi:10.1016/j.csda.2010.10.026
[42] Kiefer J, Wolfowitz J (1956) Consistency of the maximum likelihood estimator in the presence of infinitely many incidental parameters. Ann Math Stat 27(4):887-906 · Zbl 0073.14701 · doi:10.1214/aoms/1177728066
[43] Kiefer NM (1978) Discrete parameter variation: efficient estimation of a switching regression model. Econometrica 46(2):427-434 · Zbl 0408.62058 · doi:10.2307/1913910
[44] Lindsay BG (1995) Mixture models: theory, geometry and applications. NSF-CBMS regional conference series in probability and statistics, vol 5. Institute of Mathematical Statistics, Hayward, CA · Zbl 1163.62326
[45] McLachlan G, Krishnan T (2008a) The EM algorithm and extensions, 2nd edn, vol 589. Wiley, New York · Zbl 1165.62019
[46] McLachlan GJ, Krishnan T (2008b) The EM algorithm and its extensions, 2nd edn. Wiley, New York · Zbl 1165.62019 · doi:10.1002/9780470191613
[47] McLachlan GJ, Peel D (2000) Finite mixture models. Wiley, New York · Zbl 0963.62061 · doi:10.1002/0471721182
[48] McNicholas PD, Murphy TB (2008) Parsimonious Gaussian mixture models. Stat Comput 18:285-296 · doi:10.1007/s11222-008-9056-0
[49] Meng X-L (1994) On the rate of convergence of the ECM algorithm. Ann Stat 22(1):326-339 · Zbl 0803.65146 · doi:10.1214/aos/1176325371
[50] Meng X-L, van Dyk D (1997) The EM algorithm. An old folk song sung to a fast new tune. J R Stat Soc B 59(3):511-567 · Zbl 1090.62518 · doi:10.1111/1467-9868.00082
[51] Nettleton D (1999) Convergence properties of the EM algorithm in constrained spaces. Canad J Stat 27(3):639-644 · Zbl 0942.62033 · doi:10.2307/3316118
[52] O’Hagan A, White A (2016) Improved model-based clustering performance using bayes initialization averaging. Technical report, arXiv:1504.06870v4 · Zbl 1417.62177
[53] O’Hagan A, Murphy TB, Gormley C (2013) Computational aspects of fitting mixture model via the expectation-maximisation algorithm. Comput Stat Data Anal 56(12):3843-3864 · Zbl 1255.62180 · doi:10.1016/j.csda.2012.05.011
[54] Puntanen S, Styan GP, Isotalo J (2011) Matrix tricks for linear statistical models. Springer, Berlin · Zbl 1291.62014 · doi:10.1007/978-3-642-10473-2
[55] Redner RA, Walker HF (1984) Mixture densities maximum likelihood and the EM algorithm. SIAM Rev 26(2):195-239 · Zbl 0536.62021 · doi:10.1137/1026034
[56] Ritter G (2014) Cluster analysis and variable selection. CRC Press, Boca Raton · Zbl 1341.62037
[57] Rocci R, Gattone SA, Di Mari R (2016) A data driven equivariant approach to constrained Gaussian mixture modeling. Adv Data Anal Classif. doi:10.1007/s11634-016-0279-1 · Zbl 1414.62269
[58] Rousseeuw PJ, Leroy AM (2005) Robust regression and outlier detection. Wiley, New York · Zbl 0711.62030
[59] Rudin W (1976) Principles of mathematical analysis, 3rd edn. McGraw-Hill, Tokyo · Zbl 0346.26002
[60] Scrucca L, Fop M, Murphy TB, Raftery A (2016) mclust 5: clustering, classification and density estimation using Gaussian finite mixture models. R J 8(1):289-317
[61] Seo B, Kim D (2012) Root selection in normal mixture models. Comput Stat Data Anal 56:2454-2470 · Zbl 1252.62013 · doi:10.1016/j.csda.2012.01.022
[62] Subedi S, Punzo A, Ingrassia S, McNicholas PD (2013) Clustering and classification via cluster-weighted factor analyzers. Adv Data Anal Classif 7:5-40 · Zbl 1271.62137 · doi:10.1007/s11634-013-0124-8
[63] Subedi S, Punzo A, Ingrassia S, McNicholas PD (2015) Cluster-weighted \[t\] t-factor analyzers for robust model-based clustering and dimension reduction. Stat Methods Appl 24(4):623-649 · Zbl 1416.62362
[64] Tanaka K (2009) Strong consistency of the maximum likelihood estimator for finite mixtures of location-scale distributions when penalty is imposed on the ratios of the scale parameters. Scand J Stat 36(1):171-184 · Zbl 1190.62031
[65] Tanaka K, Takemura A (2006) Strong consistency of the maximum likelihood estimator for finite mixtures of location-scale distributions when the scale parameters are exponentially small. Bernoulli 12(6):1003-1017 · Zbl 1117.62025 · doi:10.3150/bj/1165269148
[66] Tarone RD, Gruenhage G (1975) A note on the uniqueness of the roots of the likelihood equations for vector-valued parameters. J Am Stat Assoc 70(352):903-904 · Zbl 0328.62018 · doi:10.1080/01621459.1975.10480321
[67] Tarone RD, Gruenhage G (1979) Corrigenda: a note on the uniqueness of the roots of the likelihood equations for vector-valued parameters. J Am Stat Assoc 74(367):744 · doi:10.2307/2287030
[68] Theobald C (1975) An inequality with application to multivariate analysis. Biometrika 62(2):461-466 · Zbl 0316.62020 · doi:10.1093/biomet/62.2.461
[69] Theobald C (1976) Corrections and amendments: an inequality with application to multivariate analysis. Biometrika 63(3):685
[70] Tipping M, Bishop CM (1999) Mixtures of probabilistic principal component mixtures of probabilistic principal component analysers. Neural Comput 11(2):443-482 · Zbl 0924.62068 · doi:10.1162/089976699300016728
[71] Titterington DM, Smith AFM, Makov UE (1985) Statistical analysis of finite mixture distributions. Wiley, New York · Zbl 0646.62013
[72] van Laarhoven PJM, Aarts EHL (1988) Simulated annealing: theory and practice. D. Reidel, Dordecht
[73] Wu CFJ (1983) On convergence properties of the EM algorithm. Ann Stat 11:95-103 · Zbl 0517.62035 · doi:10.1214/aos/1176346060
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.