On the bumpy road to the dominant mode. (English) Zbl 1226.62027

Summary: Maximum likelihood estimation in many classical statistical problems is beset by multimodality. This article explores several variations of deterministic annealing that tend to avoid inferior modes and find the dominant mode. In Bayesian settings, annealing can be tailored to find the dominant mode of the log posterior. Our annealing algorithms involve essentially trivial changes to existing optimization algorithms built on block relaxation or the EM or MM principle. Our examples include estimation with the multivariate t distribution, Gaussian mixture models, latent class analysis, factor analysis, multidimensional scaling and a one-way random effects model. In the numerical examples explored, the proposed annealing strategies significantly improve the chances for locating the global maximum.


62F15 Bayesian inference
62H25 Factor analysis and principal components; correspondence analysis
65C60 Computational problems in statistics (MSC2010)
Full Text: DOI Link


[1] Agresti, Categorical data analysis (2002) · Zbl 1018.62002
[2] Arslan, Domains of convergence for the EM algorithm: a cautionary tale in a location estimation problem, Statist. Comput. 3 pp 103– (1993)
[3] Becker, EM algorithms without missing data, Stat. Methods Med. Res. 6 pp 37– (1997)
[4] Bouguila, Clustering of count data using generalized Dirichlet multinomial distributions, IEEE Trans. Knowl. Data Eng. 20 pp 462– (2008)
[5] Dempster, Maximum likelihood from incomplete data via the EM algorithm (with discussion), J. Roy. Statist. Soc. Ser. B 39 pp 1– (1977) · Zbl 0364.62022
[6] Duan, Multiplicity of solutions in maximum likelihood factor analysis, J. Statist. Comput. Simulation 47 pp 37– (1993)
[7] Goodman, Exploratory latent structure analysis using both identifiable and unidentifiable models, Biometrika 61 pp 215– (1974) · Zbl 0281.62057
[8] Groenen, The tunneling method for global optimization in multidimensional scaling, Psychometrika 61 pp 529– (1996) · Zbl 0866.92028
[9] Heiser, Recent advances in descriptive multivariate analysis pp 157– (1995)
[10] Hunter, A tutorial on MM algorithms, Amer. Statist. 58 pp 30– (2004)
[11] Kent, A curious likelihood identity for the multivariate t-distribution, Comput. Stat. Data Anal. 41 pp 157– (1994) · Zbl 0825.62035
[12] Kirkpatrick, Optimization by simulated annealing, Science 220 pp 671– (1983) · Zbl 1225.90162
[13] Lange, Optimization (2004)
[14] Lange, Normal/independent distributions and their applications in robust regression, J. Comput. Graph. Statist. 2 pp 175– (1993)
[15] Lange, Robust statistical modeling using the t distribution, J. Amer. Statist. Assoc. 84 pp 881– (1989)
[16] Lange, Optimization transfer using surrogate objective functions (with discussion), J. Comput. Graph. Statist. 9 pp 1– (2000)
[17] De Leeuw , J 1993 Fitting distances by least squares http://preprints.stat.ucla.edu
[18] De Leeuw, Information systems and data analysis pp 308– (1994)
[19] Liu, Posterior bimodality in the balanced one-way random effects model, J. Roy. Statist. Soc. Ser. B 65 pp 247– (2003) · Zbl 1063.62100
[20] Maxwell, Recent trends in factor analysis, J. Roy. Statist. Soc. Ser. A 124 pp 49– (1961)
[21] McLachlan, The EM algorithm and extensions (2008) · Zbl 1165.62019
[22] Meng, The EM algorithm - an old folk-song sung to a fast new tune, J. Roy. Statist. Soc. Ser. B 59 pp 511– (1997) · Zbl 1090.62518
[23] Metropolis, Equations of state calculations by fast computing machines, J. Chem. Phys. 21 pp 1087– (1953)
[24] Pachter, Algebraic statistics and computational biology (2005)
[25] Press, Numerical recipes in Fortran: the art of scientific computing (1992) · Zbl 0778.65002
[26] Robert, Monte Carlo statistical methods (2004) · Zbl 1096.62003
[27] Rue, Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations (with discussion), J. Roy. Statist. Soc. Ser. B 71 pp 319– (2009) · Zbl 1248.62156
[28] Ueda, Deterministic annealing EM algorithm, Neural Netw. 11 pp 271– (1998)
[29] Wu, The MM alternative to EM, Statist. Sci. (2009) · Zbl 1329.62106
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.