×

zbMATH — the first resource for mathematics

Group coordinate descent algorithms for nonconvex penalized regression. (English) Zbl 1239.62082
Summary: We consider the problem of selecting grouped variables in linear regression and generalized linear regression models, based on penalized likelihood. A number of penalty functions have been used for this purpose, including the smoothly clipped absolute deviation (SCAD) penalty and the minimax concave penalty (MCP). These penalty functions, in comparison to the popularly used Lasso, have attractive theoretical properties such as unbiasedness and selection consistency. Although the model fitting methods using these penalties are well developed for individual variable selection, the extension to grouped variable selection is not straightforward, and the fitting can be unstable due to the nonconvexity of the penalty functions. To this end, we propose the group coordinate descent (GCD) algorithms, which extend the regular coordinate descent algorithms. These GCD algorithms are efficient, in that the computation burden only increases linearly with the number of the covariate groups.
We also show that using the GCD algorithm, the estimated parameters converge to a global minimum when the sample size is larger than the dimension of the covariates, and converge to a local minimum otherwise. In addition, we demonstrate the regions of the parameter space in which the objective function is locally convex, even though the penalty is nonconvex. In addition to group selection in the linear model, the GCD algorithms can also be extended to generalized linear regression. We present details of the extension using an example of logistic regression. The efficiency of the proposed algorithms are presented through simulation studies and a real data example, in which the MCP based and SCAD based GCD algorithms provide improved group selection results as compared to the group Lasso.

MSC:
62J05 Linear regression; mixed models
62J12 Generalized linear models (logistic models)
65C60 Computational problems in statistics (MSC2010)
90C47 Minimax problems in mathematical programming
Software:
glmnet; PDCO; S+WAVELETS
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Breiman, L., Heuristics of instability and stabilization in model selection, Ann. statist., 24, 2350-2383, (1996) · Zbl 0867.62055
[2] Bruce, A.G.; Gao, H.Y., Understanding waveshrink: variance and bias estimation, Biometrika, 83, 4, 727, (1996) · Zbl 0883.62038
[3] Chen, S.S.; Donoho, D.L.; Saunders, M.A., Atomic decomposition by basis pursuit, SIAM J. sci. comput., 20, 33-61, (1998) · Zbl 0919.94002
[4] Efron, B.; Hastie, T.; Johnstone, I.; Tibshirani, R., Least angle regression, Ann. statist., 32, 407-451, (2004) · Zbl 1091.62054
[5] Fan, J.; Li, R., Variable selection via nonconcave penalized likelihood and its oracle properties, J. amer. statist. assoc., 96, 456, 1348-1361, (2001) · Zbl 1073.62547
[6] Friedman, J.; Hastie, T.; Höing, H.; Tibshirani, R., Pathwise coordinate optimization, Ann. appl. stat., 1, 302-332, (2007) · Zbl 1378.90064
[7] Friedman, J.; Hastie, T.; Tibshirani, R., Regularization paths for generalized linear models via coordinate descent, J. stat. softw., 33, 1-22, (2010)
[8] Gao, H.Y.; Bruce, A.G., Waveshrink with firm shrinkage, Statist. sinica, 7, 855-874, (1997) · Zbl 1067.62529
[9] Huang, J.; Horowitz, J.L.; Wei, F.R., Variable selection in nonparametric models, Ann. statist., 38, 2282-2313, (2010) · Zbl 1202.62051
[10] Meier, L.; van de Geer, S.; Bühlmann, P., The group lasso for logistic regression, J. roy. statist. soc. B, 70, Part 1, 53-71, (2008) · Zbl 1400.62276
[11] Schorling, J.B.; Roach, J.; Siegel, M.; Baturka, N.; Hunt, D.E.; Guterbock, T.M.; Stewart, H.L., A trial of church-based smoking cessation interventions for rural african americans, Prev. med., 26, 92-101, (1997)
[12] Schwarz, G., Estimating the dimension of a model, Ann. statist., 6, 461-464, (1978) · Zbl 0379.62005
[13] Tibshirani, R., Regression shrinkage and selection via the lasso, J. roy. statist. soc. ser. B, 58, 267-288, (1996) · Zbl 0850.62538
[14] Wei, F.R.; Huang, J., Consistent group selection in high-dimensional linear regression, Bernoulli, 16, 4, 1369-1384, (2010) · Zbl 1207.62146
[15] Wu, T.T.; Lange, K., Coordinate descent algorithms for lasso penalized regression, Ann. appl. stat., 2, 224-244, (2008) · Zbl 1137.62045
[16] Yuan, M.; Lin, Y., Model selection and estimation in regression with grouped variables, J. roy. statist. soc. ser. B, 68, 49-67, (2006) · Zbl 1141.62030
[17] Zhang, C.H., Nearly unbiased variable selection under minimax concave penalty, Ann. statist., 38, 894-942, (2010) · Zbl 1183.62120
[18] Zhang, C.H.; Huang, J., Model selection consistency of the lasso in high-dimensional linear regression, Ann. statist., 36, 1567-1594, (2008) · Zbl 1142.62044
[19] Zou, H.; Li, R., One-step sparse estimates in nonconcave penalized likelihood models, Ann. statist., 36, 1509-1533, (2008) · Zbl 1142.62027
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.