×

Extending mixtures of multivariate \(t\)-factor analyzers. (English) Zbl 1255.62175

Summary: Model-based clustering typically involves the development of a family of mixture models and the imposition of these models upon data. The best member of the family is then chosen using some criterion and the associated parameter estimates lead to predicted group memberships, or clusterings. This paper describes the extension of the mixtures of multivariate \(t\)-factor analyzers models to include constraints on the degrees of freedom, the factor loadings, and the error variance matrices. The result is a family of six mixture models, including parsimonious models. Parameter estimates for this family of models are derived using an alternating expectation-conditional maximization algorithm and convergence is determined based on Aitken’s acceleration. Model selection is carried out using the Bayesian information criterion (BIC) and the integrated completed likelihood (ICL). This novel family of mixture models is then applied to simulated and real data where clustering performance meets or exceeds that of established model-based clustering methods. The simulation studies include a comparison of the BIC and the ICL as model selection techniques for this novel family of models. Application to simulated data with larger dimensionality is also explored.

MSC:

62H30 Classification and discrimination; cluster analysis (statistical aspects)
62H25 Factor analysis and principal components; correspondence analysis
65C60 Computational problems in statistics (MSC2010)

Software:

mclust; PGMM; BRENT; clustvarsel; R
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Banfield, J.D., Raftery, A.E.: Model-based Gaussian and non-Gaussian clustering. Biometrics 49(3), 803–821 (1993) · Zbl 0794.62034
[2] Biernacki, C., Celeux, G., Govaert, G.: Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Trans. Pattern Anal. Mach. Intell. 22(7), 719–725 (2000)
[3] Binder, D.A.: Bayesian cluster analysis. Biometrika 65, 31–38 (1978) · Zbl 0376.62007
[4] Böhning, D., Dietz, E., Schaub, R., Schlattmann, P., Lindsay, B.: The distribution of the likelihood ratio for mixtures of densities from the one-parameter exponential family. Ann. Ins. Stat. Math. 46, 373–388 (1994) · Zbl 0802.62017
[5] Brent, R.: Algorithms for Minimization without Derivatives. Prentice Hall, New Jersey (1973) · Zbl 0245.65032
[6] Celeux, G., Govaert, G.: Gaussian parsimonious clustering models. Pattern Recogn. 28, 781–793 (1995) · Zbl 05480211
[7] Dasgupta, A., Raftery, A.E.: Detecting features in spatial point processes with clutter via model-based clustering. J. Am. Stat. Assoc. 93, 294–302 (1998) · Zbl 0906.62105
[8] Day, N.E.: Estimating the components of a mixture of normal distributions. Biometrika 56, 463–474 (1969) · Zbl 0183.48106
[9] Dean, N., Raftery, A.E.: The clustvarsel package: R package version 0.2-4 (2006)
[10] Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Ser. B 39(1), 1–38 (1977) · Zbl 0364.62022
[11] Forina, M., Armanino, C., Castino, M., Ubigli, M.: Multivariate data analysis as a discriminating method of the origin of wines. Vitis 25, 189–201 (1986)
[12] Fraley, C., Raftery, A.E.: How many clusters? Which clustering methods? Answers via model-based cluster analysis. Comput. J. 41(8), 578–588 (1998) · Zbl 0920.68038
[13] Fraley, C., Raftery, A.E.: Model-based clustering, discriminant analysis, and density estimation. J. Am. Stat. Assoc. 97(458), 611–631 (2002) · Zbl 1073.62545
[14] Fraley, C., Raftery, A.E.: Enhanced software for model-based clustering, density estimation, and discriminant analysis: MCLUST. J. Classif. 20, 263–286 (2003) · Zbl 1055.62071
[15] Fraley, C., Raftery, A.E.: MCLUST: version 3 for R: normal mixture modeling and model-based clustering. Technical Report 504, Department of Statistics, University of Washington, minor revisions January 2007 and November 2007 (2006)
[16] Frühwirth-Schnatter, S.: Finite Mixture and Markov Switching Models. Springer, New York (2006) · Zbl 1108.62002
[17] Ghahramani, Z., Hinton, G.E.: The EM algorithm for factor analyzers. Tech. Rep. CRG-TR-96-1. University of Toronto, Toronto (1997)
[18] Gormley, I.C., Murphy, T.B.: A mixture of experts model for rank data with applications in election studies. Ann. Appl. Stat. 2(4), 1452–1477 (2008) · Zbl 1454.62498
[19] Hubert, L., Arabie, P.: Comparing partitions. J. Classif. 2, 193–218 (1985) · Zbl 0587.62128
[20] Hurley, C.: Clustering visualizations of multivariate data. J. Comput. Graph. Stat. 13(4), 788–806 (2004)
[21] Kass, R.E., Raftery, A.E.: Bayes factors. J. Am. Stat. Assoc. 90, 773–795 (1995) · Zbl 0846.62028
[22] Keribin, C.: Consistent estimation of the order of mixture models. Sankhyā Indian J. Stat. Ser. A 62(1), 49–66 (2000) · Zbl 1081.62516
[23] Leroux, B.G.: Consistent estimation of a mixing distribution. Ann. Stat. 20, 1350–1360 (1992) · Zbl 0763.62015
[24] Lindsay, B.G.: Mixture models: theory, geometry and applications. In: NSF-CBMS Regional Conference Series in Probability and Statistics, vol. 5. Institute of Mathematical Statistics, Hayward (1995) · Zbl 1163.62326
[25] Lopes, H.F., West, M.: Bayesian model assessment in factor analysis. Stat. Sinica 14, 41–67 (2004) · Zbl 1035.62060
[26] Lubischew, A.A.: On the use of discriminant functions in taxonomy. Biometrics 18(4), 455–477 (1962) · Zbl 0112.11602
[27] McLachlan, G.J.: The classification and mixture maximum likelihood approaches to cluster analysis. In: Handbook of Statistics, vol. 2, pp. 199–208. North-Holland, Amsterdam (1982) · Zbl 0513.62064
[28] McLachlan, G.J., Basford, K.E.: Mixture Models: Inference and Applications to Clustering. Dekker, New York (1988) · Zbl 0697.62050
[29] McLachlan, G.J., Krishnan, T.: The EM Algorithm and Extensions, 2nd edn. Wiley, New York (2008) · Zbl 1165.62019
[30] McLachlan, G.J., Peel, D.: Robust cluster analysis via mixtures of multivariate t-distributions. In: Lecture Notes in Computer Science, vol. 1451, pp. 658–666. Springer, Berlin (1998)
[31] McLachlan, G.J., Peel, D.: Finite Mixture Models. Wiley, New York (2000a) · Zbl 0963.62061
[32] McLachlan, G.J., Peel, D.: Mixtures of factor analyzers. In: Proceedings of the Seventh International Conference on Machine Learning, pp. 599–606. Morgan Kaufmann, San Francisco (2000b)
[33] McLachlan, G.J., Bean, R.W., Jones, L.B.T.: Extension of the mixture of factor analyzers model to incorporate the multivariate t-distribution. Comput. Stat. Data Anal. 51(11), 5327–5338 (2007) · Zbl 1445.62053
[34] McNicholas, P.D.: Model-based classification using latent Gaussian mixture models. J. Stat. Plan. Inference 140(5), 1175–1181 (2010) · Zbl 1181.62095
[35] McNicholas, P.D., Murphy, T.B.: Parsimonious Gaussian mixture models. Tech. Rep. 05/11, Department of Statistics, Trinity College Dublin (2005)
[36] McNicholas, P.D., Murphy, T.B.: Parsimonious Gaussian mixture models. Stat. Comput. 18, 285–296 (2008)
[37] McNicholas, P.D., Murphy, T.B.: Model-based clustering of longitudinal data. Can. J. Stat. 38(1), 153–168 (2010) · Zbl 1190.62120
[38] McNicholas, P.D., Murphy, T.B., McDaid, A.F., Frost, D.: Serial and parallel implementations of model-based clustering via parsimonious Gaussian mixture models. Comput. Stat. Data Anal. 54(3), 711–723 (2010) · Zbl 1464.62131
[39] Meng, X.L., van Dyk: The EM algorithm–an old folk song sung to a fast new tune (with discussion). J. R. Stat. Soc. Ser. B 59, 511–567 (1997) · Zbl 1090.62518
[40] Meng, X.L., Rubin, D.B.: Maximum likelihood estimation via the ECM algorithm: a general framework. Biometrika 80, 267–278 (1993) · Zbl 0778.62022
[41] R Development Core Team (2009) R: a language and environment for statistical computing: R foundation for statistical computing, Vienna, Austria. http://www.R-project.org
[42] Raftery, A.E., Dean, N.: Variable selection for model-based clustering. J. Am. Stat. Assoc. 101(473), 168–178 (2006) · Zbl 1118.62339
[43] Rand, W.M.: Objective criteria for the evaluation of clustering methods. J. Am. Stat. Assoc. 66, 846–850 (1971)
[44] Schwarz, G.: Estimating the dimension of a model. Ann. Stat. 6, 31–38 (1978) · Zbl 0379.62005
[45] Scrucca, L.: Dimension reduction for model-based clustering. Stat. Comput. (2009, in press). doi: 10.1007/s11222-009-9138-7
[46] Shoham, S.: Robust clustering by deterministic agglomeration em of mixtures of multivariate t-distributions. Pattern Recogn. 35(5), 1127–1142 (2002) · Zbl 1005.68051
[47] Swayne, D., Cook, D., Buja, A., Lang, D., Wickham, H., Lawrence, M.: (2006) GGobi Manual. Sourced from www.ggobi.org/docs/manual.pdf
[48] Tipping, T.E., Bishop, C.M.: Mixtures of probabilistic principal component analysers. Neural Comput. 11(2), 443–482 (1999a)
[49] Tipping, T.E., Bishop, C.M.: Probabilistic principal component analysers. J. R. Stat. Soc. Ser. B 61, 611–622 (1999b) · Zbl 0924.62068
[50] Wolfe, J.H.: Object cluster analysis of social areas. Master’s thesis, University of California, Berkeley (1963)
[51] Wolfe, J.H.: Pattern clustering by multivariate mixture analysis. Multiv. Behav. Res. 5, 329–350 (1970)
[52] Woodbury, M.A.: Inverting modified matrices. Statistical Research Group, Memo. Rep. No. 42, Princeton University, Princeton, New Jersey (1950)
[53] Zhao, J., Jiang, Q.: Probabilistic PCA for t distributions. Neurocomputing 69(16–18), 2217–2226 (2006) · Zbl 05184764
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.