×

A novel Bayesian approach for variable selection in linear regression models. (English) Zbl 1504.62101

Summary: A novel Bayesian approach to the problem of variable selection in multiple linear regression models is proposed. In particular, a hierarchical setting which allows for direct specification of a priori beliefs about the number of nonzero regression coefficients as well as a specification of beliefs that given coefficients are nonzero is presented. This is done by introducing a new prior for a random set which holds the indices of the predictors with nonzero regression coefficients. To guarantee numerical stability, a \(g\)-prior with an additional ridge parameter is adopted for the unknown regression coefficients. In order to simulate from the joint posterior distribution an intelligent random walk Metropolis-Hastings algorithm which is able to switch between different models is proposed. For the model transitions a novel proposal, which prefers to add a priori or empirically important predictors to the model and further tries to remove less important ones, is used. Testing the algorithm on real and simulated data illustrates that it performs at least on par and often even better than other well-established methods. Finally, it is proven that under some nominal assumptions, the presented approach is consistent in terms of model selection.

MSC:

62J05 Linear regression; mixed models
62F15 Bayesian inference
62F12 Asymptotic properties of parametric estimators
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Abramowitz, M.; Stegun, I. A., Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables (1964), Dover: Dover New York · Zbl 0171.38503
[2] Alhamzawi, R., Brq: An R package for Bayesian quantile regression (2018), Working Paper. URL https://cran.r-project.org/web/packages/Brq/Brq.pdf
[3] Alhamzawi, R.; Taha Mohammad Ali, H., The Bayesian adaptive Lasso regression, Math. Biosci., 303, 75-82 (2018) · Zbl 1409.62141
[4] Alhamzawi, R.; Yu, K.; F. B.enoit, D., Bayesian adaptive Lasso quantile regression, Stat. Model., 12, 3, 279-297 (2012) · Zbl 07257880
[5] Baragatti, M. C.; Pommeret, D., A study of variable selection using g-prior distribution with ridge parameter, Comput. Statist. Data Anal., 56, 1920-1934 (2012) · Zbl 1368.62190
[6] Bhadra, A.; Datta, J.; Polson, N. G.; Willard, B., The horseshoe+ estimator of ultra-sparse signals, Bayesian Anal., 12, 4, 1105-1131 (2017) · Zbl 1384.62079
[7] Bhattacharya, A.; Pati, D.; Pillai, N. S.; Dunson, D. B., Dirichlet-Laplace priors for optimal shrinkage, J. Amer. Statist. Assoc., 110, 512, 1479-1490 (2015) · Zbl 1373.62368
[8] Breiman, L.; Friedman, J. H., Estimating optimal transformations for multiple regression and correlation: Rejoinder, J. Amer. Statist. Assoc., 80, 391, 614-619 (1985) · Zbl 0594.62044
[9] Buehlmann, P.; Drineas, P.; Kane, M.; van der Laan, M., Handbook of Big Data (2016), Chapman & Hall/CRC · Zbl 1416.62002
[10] Carbonetto, P.; Stephens, M., Scalable variational inference for Bayesian variable selection in regression, and its accuracy in genetic association studies, Bayesian Anal., 7, 1, 73-108 (2012) · Zbl 1330.62089
[11] Carsey, T.; Harden, J., Monte Carlo Simulation and Resampling Methods for Social Science (2013), Sage Publications, Inc
[12] Carvalho, C. M.; Polson, N. G.; Scott, J. G., The horseshoe estimator for sparse signals, Biometrika, 97, 2, 465-480 (2010) · Zbl 1406.62021
[13] Chen, S.; Walker, S. G., Fast Bayesian variable selection for high dimensional linear models: Marginal solo spike and slab priors, Electron. J. Stat., 13, 1, 284-309 (2019) · Zbl 1417.62188
[14] Efron, B.; Hastie, T.; Johnstone, I.; Tibshirani, R., Least angle regression, Ann. Statist., 32, 2, 407-499 (2004) · Zbl 1091.62054
[15] Fahrmeir, L.; Kneib, T.; Lang, S.; Marx, B., Regression (2013), Springer: Springer Berlin · Zbl 1276.62046
[16] Fan, J.; Li, R., Variable selection via nonconcave penalized likelihood and its oracle properties, J. Amer. Statist. Assoc., 96, 456, 1348-1360 (2001) · Zbl 1073.62547
[17] Fernández, C.; Ley, E.; Steel, M. F., Benchmark priors for Bayesian model averaging, J. Econometrics, 100, 2, 381-427 (2001), URL http://www.sciencedirect.com/science/article/pii/S0304407600000762 · Zbl 1091.62507
[18] Fisher, C.; Mehta, P., Fast Bayesian feature selection for high dimensional linear regression in genomics via the ising approximation, Bioinformatics, 31 (2014)
[19] Foster, D. P.; Stine, R. A., Variable selection in data mining: Building a predictive model for bankruptcy, J. Amer. Statist. Assoc., 99, 303-313 (2004) · Zbl 1117.62335
[20] Friedman, J.; Hastie, T.; Tibshirani, R., Regularization paths for generalized linear models via coordinate descent, J. Stat. Softw., 33, 1, 1-22 (2010), URL http://www.jstatsoft.org/v33/i01/
[21] Gelman, A.; Rubin, D. B., Inference from iterative simulation using multiple sequences, Statist. Sci., 7, 4, 457-472 (1992) · Zbl 1386.65060
[22] Gramacy, R. B., Monomvn: Estimation for multivariate normal and student-t data with monotone missingness (2018), R package version 1.9-8. URL https://CRAN.R-project.org/package=monomvn
[23] Guan, Y.; Stephens, M., Bayesian variable selection regression for genome-wide association studies and other large-scale problems, Ann. Appl. Stat., 5, 1780-1815 (2011) · Zbl 1229.62145
[24] Gupta, M.; Ibrahim, J. G., Variable selection in regression mixture modeling for the discovery of gene regulatory networks, J. Amer. Statist. Assoc., 102, 479, 867-880 (2007) · Zbl 1469.62369
[25] Hastings, W. K., Monte Carlo sampling methods using Markov chains and their applications, Biometrika, 57, 1, 97-109 (1970) · Zbl 0219.65008
[26] Huang, A.; Liu, D., EBglmnet: Empirical Bayesian Lasso and elastic net methods for generalized linear models (2016), R package version 4.1. URL https://CRAN.R-project.org/package=EBglmnet
[27] Huang, A.; Xu, S.; hui Cai, X., Empirical Bayesian elastic net for multiple quantitative trait locus mapping, Heredity, 114, 107-115 (2015)
[28] Hurley, C., Gclus: Clustering graphics (2012), R package version 1.3.1. URL https://CRAN.R-project.org/package=gclus
[29] Jeffreys, H., Theory of Probability (1961), Oxford University Press · Zbl 0116.34904
[30] Lee, K. E.; Sha, N.; Dougherty, E. R.; Vannucci, M.; Mallick, B. K., Gene selection: a Bayesian variable selection approach, Bioinformatics, 19 1, 90-97 (2003)
[31] Leng, C.; Tran, M.-N.; Nott, D., Bayesian adaptive Lasso, Ann. Inst. Statist. Math., 66, 2, 221-244 (2014) · Zbl 1334.62130
[32] Liang, F.; Paulo, R.; Molina, G.; Clyde, M. A.; Berger, J. O., Mixtures of g priors for Bayesian variable selection, J. Amer. Statist. Assoc., 103, 481, 410-423 (2008) · Zbl 1335.62026
[33] Makalic, E.; Schmidt, D., High-dimensional Bayesian regularised regression with the bayesreg package (2016), arXiv:1611.06649v3
[34] Mitchell, T. J.; Beauchamp, J. J., Bayesian variable selection in linear regression, J. Amer. Statist. Assoc., 83, 404, 1023-1032 (1988) · Zbl 0673.62051
[35] Park, T.; Casella, G., The Bayesian lasso, J. Amer. Statist. Assoc., 103, 482, 681-686 (2008) · Zbl 1330.62292
[36] Polson, N. G.; Scott, S. L., Data augmentation for support vector machines, Bayesian Anal., 6, 1, 1-23 (2011) · Zbl 1330.62258
[37] Polson, N. G.; Scott, J. G., Local shrinkage rules, Levy processes and regularized regression, J. R. Stat. Soc. Ser. B Stat. Methodol., 74, 2, 287-311 (2012) · Zbl 1411.62209
[38] R Core Team, N. G., R: A language and environment for statistical computing (2015), R Foundation for Statistical Computing: R Foundation for Statistical Computing Vienna, Austria, URL https://www.R-project.org/
[39] Ročková, V.; George, E. I., Emvs: The EM approach to Bayesian variable selection, J. Amer. Statist. Assoc., 109, 506, 828-846 (2014) · Zbl 1367.62049
[40] Ročková, V.; George, E. I., The spike-and-slab LASSO, J. Amer. Statist. Assoc., 113, 521, 431-444 (2018) · Zbl 1398.62186
[41] Stamey, T. A.; Kabalin, J. N.; McNeal, J. E.; Johnstone, I. M.; Freiha, F. S.; Redwine, E. A.; Yang, N., Prostate specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate. II. Radical prostatectomy treated patients, J. Urol., 141 5, 1076-1083 (1989)
[42] Tibshirani, R., Regression shrinkage and selection via the lasso, J. R. Stat. Soc. Ser. B Stat. Methodol., 58, 1, 267-288 (1996) · Zbl 0850.62538
[43] Wang, M.; Sun, X.; Lu, T., Bayesian structured variable selection in linear regression models, Comput. Statist., 30, 1, 205-229 (2015) · Zbl 1342.65065
[44] West, M., Bayesian factor regression models in the “large p, small n” paradigm, (Bayesian Statistics (2003), Oxford University Press), 723-732
[45] Yi, N., A unified Markov chain Monte Carlo framework for mapping multiple quantitative trait loci, Genetics, 167, 2, 967-975 (2004)
[46] Zellner, A., On assessing prior distributions and Bayesian regression analysis with g-prior distributions, (Goel, P. K.; Zellner, A., Basic Bayesian Inference and Decision Techniques: Essays in Honor of Bruno de Finetti (1986)), 233-243 · Zbl 0655.62071
[47] Zellner, A.; Siow, A., Posterior odds ratios for selected regression hypotheses, Trab. Estad. Investig. Oper., 31, 1, 585-603 (1980) · Zbl 0457.62004
[48] Zhang, Y.; Bondell, H. D., Variable selection via penalized credible regions with Dirichlet-Laplace global-local shrinkage priors, Bayesian Anal., 13, 3, 823-844 (2018) · Zbl 1407.62272
[49] Zheng, H.; Zhang, Y., Feature selection for high dimensional data in astronomy, Adv. Space Res., 41, 12, 1960-1964 (2007)
[50] Zhou, Q.; Guan, Y., Fast model-fitting of Bayesian variable selection regression using the iterative complex factorization algorithm, Bayesian Anal., 14, 2, 573-594 (2019) · Zbl 1416.62414
[51] Zou, H., The adaptive lasso and its oracle properties, J. Amer. Statist. Assoc., 101, 476, 1418-1429 (2006) · Zbl 1171.62326
[52] Zou, H.; Hastie, T., Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B Stat. Methodol., 67, 2, 301-320 (2005) · Zbl 1069.62054
[53] Zuber, V.; Strimmer, K., Care: High-dimensional regression and CAR score variable selection (2017), R package version 1.1.10. URL https://CRAN.R-project.org/package=care
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.