×

Strong consistency of Lasso estimators. (English) Zbl 1395.62208

Summary: In this paper, we study the strong consistency and rates of convergence of the Lasso estimator. It is shown that when the error variables have a finite mean, the Lasso estimator is strongly consistent, provided the penalty parameter (say, \(\lambda_n\)) is of smaller order than the sample size (say \(n\)). We also show that this condition on \(\lambda_n\) cannot be relaxed. More specifically, we show that consistency of the Lasso estimators fail in the cases where \(\lambda_n/n\rightarrow a\) for some \(\alpha\in(0,\infty]\). For error variables with a finite \(\alpha\)th moment, \(1<\alpha <2\), we also obtain convergence rates of the Lasso estimator to the true parameter. It is noted that the convergence rates of the Lasso estimators of the non-zero components of the regression parameter vector can be worse than the corresponding least squares estimators. However, when the design matrix satisfies some orthogonality conditions, the Lasso estimators of the zero components are surprisingly accurate; The Lasso recovers the zero components exactly, for large \(n\), almost surely.

MSC:

62J07 Ridge regression; shrinkage estimators (Lasso)
60F15 Strong limit theorems
62E20 Asymptotic distribution theory in statistics
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Athreya, K. B. and Lahiri, S. N. (2006). Measure theory and Probability theory. Springer Texts in Statistics, Springer, New York. · Zbl 1125.60001
[2] Bickel, P. J., Ritov, Y. and Tsybakov, A. B. (2009). Simultaneous analysis of lasso and dantzig selector. Ann. Statist., 37, 1705–1732. · Zbl 1173.62022 · doi:10.1214/08-AOS620
[3] Drygas, H. (1976). Weak and strong consistency of the least squares estimators in regression models. Z. Wahrscheinlichkeitstheorie und Verw. Gebiete, 34, 119–127. · Zbl 0307.62047 · doi:10.1007/BF00535679
[4] Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression. Ann. Statist., 32, 407–499. With discussion, and a rejoinder by the authors. · Zbl 1091.62054 · doi:10.1214/009053604000000067
[5] Fu, W. J. (1998). Penalized regressions: the bridge versus the lasso. J. Comput. Graph. Statist., 7, 397–416.
[6] Huang, J., Horowitz, J. L. and Ma, S. (2008). Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Statist., 36, 587–613. · Zbl 1133.62048 · doi:10.1214/009053607000000875
[7] Knight, K. and Fu, W. (2000). Asymptotics for lasso-type estimators. Ann. Statist., 28, 1356–1378. · Zbl 1105.62357 · doi:10.1214/aos/1015957397
[8] Lai, T. L., Robbins, H. and Wei, C. Z. (1978). Strong consistency of least squares estimates in multiple regression. Proc. Nat. Acad. Sci. U.S.A., 75, 3034–3036. · Zbl 0386.62051 · doi:10.1073/pnas.75.7.3034
[9] Lounici, K. (2008). Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators. Electron. J. Stat., 2, 90–102. · Zbl 1306.62155 · doi:10.1214/08-EJS177
[10] Meinshausen, N. and Bühlmann, P. (2006). High-dimensional graphs and variable selection with the lasso. Ann. Statist., 34, 1436–1462. · Zbl 1113.62082 · doi:10.1214/009053606000000281
[11] Meinshausen, N. and Yu, B. (2009). Lasso-type recovery of sparse representations for high-dimensional data. Ann. Statist., 37, 246–270. · Zbl 1155.62050 · doi:10.1214/07-AOS582
[12] Osborne, M. R., Presnell, B. and Turlach, B. A. (2000). On the LASSO and its dual. J. Comput. Graph. Statist., 9, 319–337.
[13] Rao, C. R. (1973). Linear statistical inference and its applications. John Wiley &amp; Sons, New York London-Sydney, second edition. Wiley Series in Probability and Mathematical Statistics. · Zbl 0256.62002
[14] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc., Ser. B, 58, 267–288. · Zbl 0850.62538
[15] Wainwright, M. J. (2006). Sharp thresholds for high-dimensional and noisy recovery of sparsity. Technical report, Dept. of Statistics, UC Berkeley. Preprint available at http://arxiv.org/abs/math/0605740 .
[16] Zhang, C.-H. and Huang, J. (2008). The sparsity and bias of the lasso selection in high-dimensional linear regression. Ann. Statist., 36, 1567–1594. · Zbl 1142.62044 · doi:10.1214/07-AOS520
[17] Zhao, P. and Yu, B. (2006). On model selection consistency of Lasso. J. Mach. Learn. Res., 7, 2541–2563. · Zbl 1222.62008
[18] Zou, H. (2006). The adaptive lasso and its oracle properties. J. Amer. Statist. Assoc., 101, 1418–1429. · Zbl 1171.62326 · doi:10.1198/016214506000000735
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.