Bayesian effect fusion for categorical predictors. (English) Zbl 1416.62224

Summary: We propose a Bayesian approach to obtain a sparse representation of the effect of a categorical predictor in regression type models. As this effect is captured by a group of level effects, sparsity cannot only be achieved by excluding single irrelevant level effects or the whole group of effects associated to this predictor but also by fusing levels which have essentially the same effect on the response. To achieve this goal, we propose a prior which allows for almost perfect as well as almost zero dependence between level effects a priori. This prior can alternatively be obtained by specifying spike and slab prior distributions on all effect differences associated to this categorical predictor. We show how restricted fusion can be implemented and develop an efficient MCMC (Markov chain Monte Carlo) method for posterior computation. The performance of the proposed method is investigated on simulated data and we illustrate its application on real data from EU-SILC (European Union Statistics on Income and Living Conditions).


62G08 Nonparametric regression and quantile regression
62F15 Bayesian inference
62P20 Applications of statistics to economics


glasso; CasANOVA
Full Text: DOI arXiv Euclid


[1] Binder, D. A. (1978). “Bayesian Cluster Analysis.” Biometrika, 65: 31–38. · Zbl 0376.62007
[2] Bondell, H. D. and Reich, B. J. (2009). “Simultaneous Factor Selection and Collapsing Levels in ANOVA.” Biometrics, 65: 169–177. · Zbl 1159.62048
[3] Chen, R.-B., Chu, C.-H., Yuan, S., and Wu, Y. N. (2016). “Bayesian Sparse Group Selection.” Journal of Computational and Graphical Statistics, 25: 665–683.
[4] Chipman, H. (1996). “Bayesian Variable Selection with Related Predictors.” The Canadian Journal of Statistics, 1: 17–36. · Zbl 0849.62032
[5] Dellaportas, P. and Tarantola, C. (2005). “Model Determination for Categorical Data with Factor Level Merging.” Journal of Royal Statistical Society, Series B, 67: 269–283. · Zbl 1069.62049
[6] Fahrmeir, L., Kneib, T., and Konrath, S. (2010). “Bayesian Regularisation in Structured Additive Regression: A Unifying Perspective on Shrinkage, Smoothing and Predictor Selection.” Statistics and Computing, 20: 203–219.
[7] Fritsch, A. and Ickstadt, K. (2009). “Improved Criteria for Clustering Based on the Posterior Similarity Matrix.” Bayesian Analysis, 4: 367–392. · Zbl 1330.62249
[8] George, E. and McCulloch, R. (1997). “Approaches for Bayesian Variable Selection.” Statistica Sinica, 7: 339–373. · Zbl 0884.62031
[9] Gertheiss, J. and Tutz, G. (2009). “Penalized Regression with Ordinal Predictors.” International Statistical Review, 345–365.
[10] Gertheiss, J. and Tutz, G. (2010). “Sparse Modelling of Categorical Explanatory Variables.” The Annals of Applied Statistics, 4: 2150–2180. · Zbl 1220.62092
[11] Griffin, J. and Brown, P. J. (2010). “Inference with Normal-Gamma Prior Distributions in Regression Problems.” Bayesian Analysis, 5: 171–188. · Zbl 1330.62128
[12] Ishwaran, H. and Rao, S. J. (2005). “Spike and Slab Variable Selection; Frequentist and Bayesian Strategies.” Annals of Statistics, 33: 730–773. · Zbl 1068.62079
[13] Kyung, M., Gill, J., Ghosh, M., and Casella, G. (2010). “Penalized Regression, Standard Errors, and Bayesian Lassos.” Bayesian Analysis, 5(2): 369–412. · Zbl 1330.62289
[14] Lau, J. W. and Green, P. J. (2007). “Bayesian Model Based Clustering Procedures.” Journal of Computational and Graphical Statistics, 16: 526–558.
[15] Lee, K.-J. and Chen, R.-B. (2015). “BSGS: Bayesian Sparse Group Selection.” The R Journal, 7(2): 122–132.
[16] Liu, F., Chakraborty, S., Li, F., Liu, Y., and Lozano, A. C. (2014). “Bayesian Regularization via Graph Laplacian.” Bayesian Analysis, 9(2): 449–474. · Zbl 1327.62152
[17] Malsiner-Walli, G., Pauger, D., and Wagner, H. (2018). “Effect Fusion Using Model-Based Clustering.” Statistical Modelling, 18(2): 175–196
[18] Mitchell, T. and Beauchamp, J. J. (1988). “Bayesian Variable Selection in Linear Regression.” Journal of the American Statistical Association, 83: 1023–1032. · Zbl 0673.62051
[19] Park, T. and Casella, G. (2008). “The Bayesian Lasso.” Journal of the American Statistical Association, 103(482): 681–686. · Zbl 1330.62292
[20] Pauger, D. and Wagner, H. (2018). “Supplementary Material of “Bayesian Effect Fusion for Categorical Predictors”.” Bayesian Analysis.
[21] Pauger, D., Wagner, H., and Malsiner-Walli, G. (2016). “effectFusion: Bayesian Effect Fusion for Categorical Predictors.” http://www.R-project.org/.
[22] Raman, S., Fuchs, T. J., Wild, P. J., Dahl, E., and Roth, V. (2009). “The Bayesian Group-Lasso for Analyzing Contingency Tables.” In Proceedings of the 26th Annual International Conference on Machine Learning. ICML 2009, Montreal.
[23] Scheipl, F., Fahrmeir, L., and Kneib, T. (2012). “Spike-and-Slab Priors for Function Selection in Structured Additive Regression Models.” Journal of the American Statistical Association, 107(500): 1518–1532. · Zbl 1258.62082
[24] Simon, N., Friedman, J., Hastie, T., and Tibshirani, R. (2013). “A Sparse-Group Lasso.” Journal of Computational and Graphical Statistics, 22(2): 231–245.
[25] Sun, D., Tsutakawa, R. K., and He, Z. (2001). “Propriety of Posteriors with Improper Priors in Hierarchical Linear Mixed Models.” Statistica Sinica, 11: 77–95. · Zbl 1057.62525
[26] Tibshirani, R. (1996). “Regression Shrinkage and Selection via the Lasso.” Journal of Royal Statistical Society, Series B, 58(1): 267–288. · Zbl 0850.62538
[27] Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., and Kneight, K. (2005). “Sparsity and Smoothness via the Fused Lasso.” Journal of Royal Statistical Society, Series B, 67(1): 91–108. · Zbl 1060.62049
[28] Tutz, G. and Berger, M. (2018). “Tree-Structured Clustering in Fixed Effects Models.” Journal of Computational and Graphical Statistics, published online. · Zbl 1416.62364
[29] Tutz, G. and Gertheiss, J. (2016). “Regularized Regression for Categorical Data.” Statistical Modelling, 16(3): 161–200.
[30] Yuan, M. and Lin, Y. (2006). “Model Selection and Estimation in Regression with Grouped Variables.” Journal of Royal Statistical Society, Series B, 68: 49–67. · Zbl 1141.62030
[31] Zou, H. and Hastie, T. (2005). “Regularization and Variable Selection via the Elastic Net.” Journal of Royal Statistical Society, Series B, 67: 301–320. · Zbl 1069.62054
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.