×

Sparse and robust estimation with ridge minimax concave penalty. (English) Zbl 07769004

Summary: Feature selection is an important procedure that is used in data mining to extract valuable information from large quantities of data. Existing penalization methods use a single penalty function to select important features. However, these methods do not yield sufficiently accurate predictions and selection outcomes. Therefore, construction of a concise and efficient prediction model would be beneficial. In this study, we propose a novel penalty function using a ridge and minimax concave penalty to overcome the limitations of individual penalty functions. Furthermore, we introduce a robust penalized feature selection method with Huber loss function, which is implemented by a local approximation algorithm. The theoretical properties of the algorithm have been described. Simulated and real-world data analyses are used to demonstrate the efficacy of the proposed method.

MSC:

62-XX Statistics
68-XX Computer science
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Grubbs, Frank E., Procedures for detecting outlying observations in samples, Technometrics, 11, 1, 1-21 (1969)
[2] Jr Branham, R. L., Alternatives to least squares, Astron. J., 87, 6, 928-937 (1982)
[3] Mosteller, F.; Tukey, J. W., Data analysis and regression: a second course in statistics (1977), Addison-Wesley
[4] Rousseeuw, Peter J., Least median of squares regression, J. Am. Stat. Assoc., 79, 388, 871-880 (1984) · Zbl 0547.62046
[5] Rousseeuw, Peter; Croux, Christophe, Alternatives to median absolute deviation, J. Am. Stat. Assoc., 88, 1273-1283 (1993) · Zbl 0792.62025
[6] Huber, P. J., Robust regression: asymptotics, conjectures and monte carlo, Ann. Stat., 1, 799-821 (1973) · Zbl 0289.62033
[7] Holder, R.; Mosteller, Frederick; Tukey, John, Data analysis and regression, Appl. Stat., 28, 177 (1979)
[8] Andrews, D. F., A robust method for multiple linear regression, Technometrics, 16, 4, 523-531 (1974) · Zbl 0294.62082
[9] Koenker, Roger W.; Bassett, Gilbert, Regression quantile, Econometrica, 46, 1, 33-50 (1978) · Zbl 0373.62038
[10] Roger Koenker, Pin Ng, A frisch-newton algorithm for sparse quantile regression, Acta Math. Appl. Sin. (Engl. Ser.) (02) (2005) 51-62. · Zbl 1097.62028
[11] Farcomeni, Alessio; Geraci, Marco, Multistate quantile regression models, Stat. Med., 39, 45-56 (2019)
[12] Yuan, Chao; Yang, Liming; Sun, Ping, Correntropy-based metric for robust twin support vector machine, Inf. Sci., 545, 82-101 (2021) · Zbl 1475.68307
[13] He, Yicong; Wang, Fei; Wang, Shiyuan; Cao, Jiuwen; Chen, Badong, Maximum correntropy adaptation approach for robust compressive sensing reconstruction, Inf. Sci., 480, 381-402 (2019) · Zbl 1458.94098
[14] Li, Ming; Huang, Changqin; Wang, Dianhui, Robust stochastic configuration networks with maximum correntropy criterion for uncertain data regression, Inf. Sci., 473, 73-86 (2019) · Zbl 1450.62123
[15] Zhou, Peng; Xuegang, Xuegang; Li, Peipei; Wu, Xindong, Online streaming feature selection using adapted neighborhood rough set, Inf. Sci., 481, 258-279 (2019)
[16] Sheikhpour, Razieh; Sarram, Mehdi Agha; Sheikhpour, Elnaz, Semi-supervised sparse feature selection via graph laplacian based scatter matrix for regression problems, Inf. Sci., 468, 14-28 (2018) · Zbl 1450.62080
[17] Frank, Lldiko E.; Friedman, Jerome H., A statistical view of some chemometrics regression tools, Technometrics, 35, 2, 109-135 (1993) · Zbl 0775.62288
[18] Tibshirani, Robert, Regression shrinkage and selection via the lasso, J. R. Stat. Soc., 58, 1, 267-288 (1996) · Zbl 0850.62538
[19] Fan, Jianqing; Li, Runze, Variable selection via nonconcave penalized likelihood and its oracle properties, J. Am. Stat. Assoc., 96, 456, 1348-1360 (2001) · Zbl 1073.62547
[20] Zou, Hui; Hastie, Trevor, Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B, 67, 768 (2005) · Zbl 1069.62054
[21] Zou, Hui, The adaptive lasso and its oracle properties, J. Am. Stat. Assoc., 101, 476, 1418-1429 (2006) · Zbl 1171.62326
[22] Zhang, Cun Hui, Nearly unbiased variable selection under minimax concave penalty, Ann. Stat., 38, 2, 894-942 (2010) · Zbl 1183.62120
[23] Selesnick, Ivan, Sparse regularization via convex analysis, IEEE Trans. Signal Process., 65, 17, 4481-4494 (2017) · Zbl 1414.94545
[24] Wang, Hansheng; Li, Guodong; Jiang, Guohua, Robust regression shrinkage and consistent variable selection through the lad-lasso, J. Business Econ. Stat., 25, 347-355 (2007)
[25] Arslan, Olcay, Weighted lad-lasso method for robust parameter estimation and variable selection in regression, Comput. Stat. Data Anal., 56, 6, 1952-1965 (2012) · Zbl 1243.62029
[26] Yang, Hu.; Yang, Jing, The adaptive l1-penalized lad regression for partially linear single-index models, J. Stat. Plann. Inference, 151-152, 73-89 (2014) · Zbl 1288.62065
[27] Wang, Mingqiu; Song, Lixin; Tian, Guo-liang, Scad-penalized least absolute deviation regression in high-dimensional models, Commun. Stat., 44, 12, 2452-2472 (2015) · Zbl 1328.62111
[28] Bin Li, Qingzhao Yu, Robust and sparse bridge regression, Stat. Interface 4 (2009) 481-491. · Zbl 1245.62092
[29] Lambert-Lacroix, Sophie; Zwald, Laurent, Robust regression through the huber’s criterion and adaptive lasso penalty, Electron. J. Stat., 5, 1015-1053 (2011) · Zbl 1274.62467
[30] Lamarche, Carlos, Robust penalized quantile regression estimation for panel data, J. Econometr., 157, 2, 396-408 (2010) · Zbl 1431.62161
[31] Belloni, Alexandre; Chernozhukov, Victor, L1-penalized quantile regression in high-dimensional sparse models, Ann. Stat., 39, 1, 82-130 (2011) · Zbl 1209.62064
[32] Wang, Lan; Wu, Yichao; Li, Runze, Quantile regression for analyzing heterogeneity in ultra-high dimension, J. Am. Stat. Assoc., 107, 497, 214-222 (2012) · Zbl 1328.62468
[33] Yi, Congrui; Huang, Jian, Semismooth newton coordinate descent algorithm for elastic-net penalized huber loss regression and quantile regression, J. Comput. Graph. Stat., 26, 3, 547-557 (2017)
[34] Yuwen, Gu.; Fan, Jun; Kong, Lingchen; Ma, Shiqian; Zou, Hui, Admm for high-dimensional sparse penalized quantile regression, Technometrics, 60, 3, 319-331 (2018)
[35] Ding, Xianwen; Chen, Jiandong; Chen, Xueping, Regularized quantile regression for ultrahigh-dimensional data with nonignorable missing responses, Metrika, 09 (2019) · Zbl 1442.62082
[36] Liu, Yongxin; Zeng, Peng; Lu, Lin, Generalized l1-penalized quantile regression with linear constraints, Comput. Stat. Data Anal., 142, Article 106819 pp. (2020) · Zbl 1507.62118
[37] Feng, Lei; Sun, Huaijiang; Zhu, Jun, Robust image compressive sensing based on half-quadratic function and weighted schatten-p norm, Inf. Sci., 477, 265-280 (2019) · Zbl 1442.94009
[38] Liu, Xinxin; Zhou, Yucan; Zhao, Hong, Robust hierarchical feature selection driven by data and knowledge, Inf. Sci. (2020)
[39] Sang, Xiaoshuang; Lu, Hong; Zhao, Qinghua; Zhang, Faen; Jianfeng, Lu., Nonconvex regularizer and latent pattern based robust regression for face recognition, Inf. Sci., 547, 384-403 (2021)
[40] Jiang, He; Luo, Shihua; Dong, Yao, Simultaneous feature selection and clustering based on square root optimization, Eur. J. Oper. Res., 289, 1, 214-231 (2021) · Zbl 1487.62086
[41] Hoerl, Arthur E.; Kennard, Robert W., Ridge regression: biased estimation for nonorthogonal problems, Technometrics, 12, 1, 55-67 (1970) · Zbl 0202.17205
[42] Zhang, Cun Hui; Zhang, Tong, A general theory of concave regularization for high-dimensional sparse estimation problems, Stat. Sci., 27, 4, 576-593 (2012) · Zbl 1331.62353
[43] Bickel, Peter J.; Ritov, Ya’acov; Tsybakov, Alexandre B., Simultaneous analysis of lasso and dantzig selector, Ann. Stat., 37, 4, 1705-1732 (2009) · Zbl 1173.62022
[44] Peter J. Bickel, Ya’acov Ritov, Alexandre B. Tsybakov, et al., Hierarchical selection of variables in sparse high-dimensional regression, in: Borrowing strength: theory powering applications-a Festschrift for Lawrence D. Brown, Institute of Mathematical Statistics, 2010, pp. 56-69.
[45] Wadsworth, Jennifer L.; Tawn, Jonathan A., Asymptotic properties for combined l1 and concave regularization, Biometrika, 1, 1, 57-70 (2014) · Zbl 1285.62074
[46] Zhao, Tuo; Liu, Han; Zhang, Tong, Pathwise coordinate optimization for sparse learning: algorithm and theory, Ann. Stat., 46, 1, 180-218 (2018) · Zbl 1416.62413
[47] Fan, Jianqing; Fan, Yingying; Barut, Emre, Adaptive robust variable selection, Ann. Stat., 42, 1, 324-351 (2014) · Zbl 1296.62144
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.