×

Parsimonious additive models. (English) Zbl 1161.62354

Summary: A new method for function estimation and variable selection, specifically designed for additive models fitted by cubic splines is proposed. This new method involves regularizing additive models using the \(l_{1}\)-norm, which generalizes the lasso to the nonparametric setting. As in the linear case, it shrinks coefficients and produces some coefficients that are exactly zero. It gives parsimonious models, selects significant variables, and reveals nonlinearities in the effects of predictors. Two strategies for finding a parsimonious additive model solution are proposed. Both algorithms are based on a fixed point algorithm, combined with a singular value decomposition that considerably reduces computation. The empirical behavior of parsimonious additive models is compared to the adaptive backfitting BRUTO algorithm. The results allow to characterize the domains in which our approach is effective: it performs significantly better than BRUTO when model estimation is challenging. An implementation of this method is illustrated using real data from the Cophar 1 ANRS 102 trial. Parsimonious additive models are applied to predict the indinavir plasma concentration in HIV patients. Results suggest that this new method is a promising technique for the research and application areas.

MSC:

62G08 Nonparametric regression and quantile regression
65C60 Computational problems in statistics (MSC2010)
62P10 Applications of statistics to biology and medical sciences; meta analysis
PDFBibTeX XMLCite
Full Text: DOI Link

References:

[1] Avalos, M.; Grandvalet, Y.; Ambroise, C., Regularization methods for additive models, (Berthold, M. R.; Lenz, H. J.; Bradley, E.; Kruse, R.; Borgelt, C., Fifth International Symposium on Intelligent Data Analysis, Lecture Notes Computer Science (2003), Springer: Springer Berlin), 509-520
[2] Bacchetti, P.; Quale, C., Generalized additive models with interval-censored data and time-varying covariates: application to human immunodeficiency virus infection in hemophiliacs, Biometrics, 58, 2, 443-447 (2002) · Zbl 1210.62040
[3] Bakin, S., 1999. Adaptive regression and model selection in data mining problems. Ph.D. Thesis, School of Mathematical Sciences, The Australian National University, Canberra.; Bakin, S., 1999. Adaptive regression and model selection in data mining problems. Ph.D. Thesis, School of Mathematical Sciences, The Australian National University, Canberra.
[4] Beck, N.; Jackman, S., Beyond linearity by default: generalized additive models, Amer. J. Political Sci., 42, 596-627 (1998)
[5] Breiman, L., Heuristics of instability and stabilization in model selection, Ann. Statist., 24, 6, 2350-2383 (1996) · Zbl 0867.62055
[6] Brendel, K., Legrand, M., Taburet, A.M., Baron, G., Goujard, C., Mentre, F., Cophar 1-ANRS 102 Trial Group, 2005. Population pharmacokinetic analysis of indinavir in HIV-infected patient treated with a stable antiretroviral therapy. Fund. Clin. Pharmacol. 19(3), 373-383.; Brendel, K., Legrand, M., Taburet, A.M., Baron, G., Goujard, C., Mentre, F., Cophar 1-ANRS 102 Trial Group, 2005. Population pharmacokinetic analysis of indinavir in HIV-infected patient treated with a stable antiretroviral therapy. Fund. Clin. Pharmacol. 19(3), 373-383.
[7] Brumback, B. A.; Ruppert, D.; Wand, M. P., Comment on “Variable selection and function estimation in additive nonparametric regression using a data-based prior” by Shively, T.S. and Khon, R. and Wood, S, J. Amer. Statist. Assoc., 94, 447, 794-797 (1999)
[8] Cantoni, E.; Hastie, T. J., Degrees of freedom tests for smoothing splines, Biometrika, 89, 251-263 (2002) · Zbl 1019.62038
[9] Chambers, J. M.; Hastie, T. J., Statistical Models in S. Computer Science Series (1993), Chapman & Hall: Chapman & Hall London
[10] Chen, R.; Härdle, W.; Linton, O. B.; Severance-Lossin, E., Nonparametric estimation of additive separable regression models, (Härdle, W.; Schimek, M. G., Statistical Theory and Computational Aspects of Smoothing: Proceedings of the COMPSTAT’94 Satellite Meeting (1996), Heidelberg: Heidelberg Physica-Verlag), 247-265
[11] Chen, S., Donoho, D., Saunders, M., 1995. Atomic decomposition by basis pursuit. Technical Report 479, Department of Statistics, Stanford University.; Chen, S., Donoho, D., Saunders, M., 1995. Atomic decomposition by basis pursuit. Technical Report 479, Department of Statistics, Stanford University. · Zbl 0919.94002
[12] Chen, Z., Fitting multivariate regression functions by interaction spline models, J. Roy. Statist. Soc. Ser. B, 55, 2, 473-491 (1993) · Zbl 0783.62029
[13] Dominici, F.; McDermott, A.; Zeger, S. L.; Samet, J. M., On the use of generalized additive models in time-series studies of air pollution and health, Amer. J. Epidemiol., 156, 3, 193-203 (2002)
[14] Efron, B.; Hastie, T.; Johnstone, I.; Tibshirani, R., Least angle regression, Ann. Statist., 32, 2, 407-499 (2004) · Zbl 1091.62054
[15] Fu, W. J., Penalized regression: the bridge versus the lasso, J. Comput. Graph. Statist., 7, 3, 397-416 (1998)
[16] González-Manteiga, W.; Quintela-del Río, A.; Vieu, P., A note on variable selection in nonparametric regression with dependent data, Statist. Probab. Lett., 57, 3, 259-268 (2002) · Zbl 0996.62036
[17] Grandvalet, Y., Least absolute shrinkage is equivalent to quadratic penalization, (Niklasson, L.; Bodén, M.; Ziemske, T., ICANN’98 Perspectives in Neural Computing, vol. 1 (1998), Springer: Springer Berlin), 201-206
[18] Grandvalet, Y.; Canu, S., Outcomes of the equivalence of adaptive ridge with least absolute shrinkage, (Kearns, M. S.; Solla, S. A.; Cohn, D. A., Advances in Neural Information Processing Systems, vol. 11 (1998), MIT Press: MIT Press Cambridge, MA), 445-451
[19] Gu, C.; Wahba, G., Minimizing GCV/GML scores with multiple smoothing parameters via the newton method, SIAM J. Sci. Statist. Comput., 12, 383-398 (1991) · Zbl 0727.65009
[20] Gunn, S. R.; Kandola, J. S., Structural modeling with sparse kernels, Mach. Learn., 10, 581-591 (2002) · Zbl 0998.68119
[21] Guyon, I.; Elisseeff, A., An introduction to variable and feature selection, J. Mach. Learn. Res., 3, Special Issue on Variable/Feature Selection, 1157-1182 (2003) · Zbl 1102.68556
[22] Härdle, W.; Korostelev, A., Search for significant variables in nonparametric additive regression, Biometrika, 83, 3, 541-549 (1996) · Zbl 0866.62016
[23] Hastie, T. J.; Tibshirani, R. J., Generalized Additive Models, Monographs on Statistics and Applied Probability, vol. 43 (1990), Chapman & Hall: Chapman & Hall London · Zbl 0747.62061
[24] Hastie, T. J.; Tibshirani, R. J.; Friedman, J., The Elements of Statistical Learning. Data Mining, Inference and Prediction (2001), Springer Series in Statistics: Springer Series in Statistics Springer, New York · Zbl 0973.62007
[25] Hurvich, C. M.; Simonoff, J. S.; Tsai, C. L., Smoothing parameter selection in nonparametric regression using an improved akaike information criteria, J. Roy. Statist. Soc. Ser. B, 60, 2, 271-293 (1998) · Zbl 0909.62039
[26] Lin, Y., Zhang, H.H., 2002. Component selection and smoothing in smoothing spline analysis of variance models. Technical Report 1072r, University of Winconsin—Madison and North Carolina State University (Rev janvier 2003).; Lin, Y., Zhang, H.H., 2002. Component selection and smoothing in smoothing spline analysis of variance models. Technical Report 1072r, University of Winconsin—Madison and North Carolina State University (Rev janvier 2003).
[27] Marx, B. D.; Eilers, P. H.C., Direct generalized additive modeling with penalized likelihood, Comput. Statist. Data Anal., 28, 193-209 (1998) · Zbl 1042.62580
[28] Opsomer, J. D.; Ruppert, D., A fully automated bandwidth selection method for fitting additive models, J. Multivariate Anal., 73, 166-179 (1998) · Zbl 0953.62034
[29] Osborne, M. R.; Presnell, B.; Turlach, B. A., On the lasso and its dual, J. Comput. Graph. Statist., 9, 2, 319-337 (2000)
[30] Roth, V., Sparse kernel regressors, (Dorfner, G.; Bischof, H.; Hornik, K., Artificial Neural Networks-ICANN 2001, Lecture Notes in Computer Science, vol. 2130 (2001), Springer: Springer Berlin), 339-346 · Zbl 1001.68790
[31] Ruppert, D.; Wand, M. P.; Carroll, R. J., Semiparametric Regression Cambridge Series on Statistical and Probabilistic Mathematics, vol. 12 (2003), Cambridge University Press: Cambridge University Press Cambridge
[32] Shi, P.; Tsai, C.-L., Semiparametric regression model selections, J. Statist. Plann. Inference, 77, 1, 119-139 (1999) · Zbl 1070.62506
[33] Shively, T. S.; Khon, R.; Wood, S., Variable selection and function estimation in additive nonparametric regression using a data-based prior, J. Amer. Statist. Assoc., 99, 447, 777-806 (1999) · Zbl 0994.62033
[34] Smith, M.; Kohn, R., Nonparametric regression using Bayesian variable selection, J. Econom., 75, 2, 317-343 (1996) · Zbl 0864.62025
[35] Stone, M., Cross-validatory choice and assessment of statistical predictions, J. Roy. Statist. Soc. Ser. B, 36, 1, 111-147 (1974) · Zbl 0308.62063
[36] Tibshirani, R. J., Regression shrinkage and selection via the lasso, J. Roy. Statist. Soc. Ser. B, 58, 1, 267-288 (1996) · Zbl 0850.62538
[37] Wahba, G., Spline models for observational data. Regional Conference Series in Applied Mathematics, vol. 59 (1990), SIAM: SIAM Philadelphia, PA · Zbl 0813.62001
[38] Walker, E.; Wright, S. P., Comparing curves using additive models, J. Qual. Technol., 34, 1, 118-129 (2002)
[39] Wood, S. N., Modelling and smoothing parameter estimation with multiple quadratic penalties, J. Roy. Statist. Soc. Ser. B, 62, 2, 413-428 (2000)
[40] Zhang, H. H.; Wahba, G.; Lin, Y.; Voelker, M.; Ferris, M.; Klein, R.; Klein, B., Variable selection and model building via likelihood basis pursuit, J. Amer. Statist. Assoc., 99, 659-672 (2004) · Zbl 1117.62459
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.