×

Penalized wavelets: embedding wavelets into semiparametric regression. (English) Zbl 1271.62089

Summary: We introduce the concept of penalized wavelets to facilitate seamless embedding of wavelets into semiparametric regression models. In particular, we show that penalized wavelets are analogous to penalized splines; the latter being the established approach to function estimation in semiparametric regression. They differ only in the type of penalization that is appropriate. This fact is not borne out by the existing wavelet literature, where the regression modelling and fitting issues are overshadowed by computational issues such as efficiency gains afforded by the discrete wavelet transform and partially obscured by a tendency to work in the wavelet coefficient space. With penalized wavelet structure in place, we then show that fitting and inference can be achieved via the same general approaches used for penalized splines: penalized least squares, maximum likelihood and best prediction within a frequentist mixed model framework, and Markov chain Monte Carlo and mean field variational Bayes within a Bayesian framework. Penalized wavelets are also shown have a close relationship with wide data (“\(p\gg n\)”) regression and benefit from ongoing research on that topic.

MSC:

62G08 Nonparametric regression and quantile regression
42C40 Nontrigonometric harmonic analysis involving wavelets and other special systems
62F05 Asymptotic properties of parametric tests
PDFBibTeX XMLCite
Full Text: DOI Euclid

References:

[1] Albert, J.H. and Chib, S. (1993). Bayesian analysis of binary and polychotomous response data., Journal of the American Statistical Association , 88 , 669-679. · Zbl 0774.62031 · doi:10.2307/2290350
[2] Antoniadis, A., Bigot, J. and Gijbels, I. (2007). Penalized wavelet monotone regression., Statistics and Probability Letters , 77 , 1608-1621. · Zbl 1127.62034 · doi:10.1016/j.spl.2007.03.041
[3] Antoniadis, A. and Fan, J. (2001). Regularization of wavelet approximations (with discussion)., Journal of the American Statistical Association , 96 , 939-967. · Zbl 1072.62561 · doi:10.1198/016214501753208942
[4] Antoniadis, A. and Leblanc, F. (2000). Nonparametric wavelet regression for binary response., Statistics , 34 , 183-213. · Zbl 0955.62041 · doi:10.1080/02331880008802713
[5] Attias, H. (1999). Inferring parameters and structure of latent variable models by variational Bayes., Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence , 21-30.
[6] Aykroyd, R.G. and Mardia, K.V. (2003). A wavelet approach to shape analysis for spinal curves., Journal of Applied Statistics , 30 , 605-623. · Zbl 1121.62321 · doi:10.1080/0266476032000053718
[7] Berry, S.M., Carroll, R.J. and Ruppert, D. (2002). Bayesian smoothing and regression splines for measurement error problems., Journal of the American Statistical Association , 97 , 160-169. · Zbl 1073.62524 · doi:10.1198/016214502753479301
[8] Bishop, C.M. (2006)., Pattern Recognition and Machine Learning. New York: Springer. · Zbl 1107.68072
[9] Breheny, P. (2011). ncvreg 2.3. Regularization paths for SCAD- and MCP-penalized regression models. R package.,
[10] Brumback, B.A. and Rice, J.A. (1998). Smoothing spline models for the analysis of nested and crossed samples of curves (with discussion)., Journal of the American Statistical Association , 93 , 961-994. · Zbl 1064.62515 · doi:10.2307/2669837
[11] Buja, A., Hastie, T. and Tibshirani, R. (1989). Linear smoothers and additive models., The Annals of Statistics , 17 , 453-510. · Zbl 0689.62029 · doi:10.1214/aos/1176347115
[12] Carvalho, C.M., Polson, N.G. and Scott, J.G. (2010). The horseshoe estimator for sparse signals., Biometrika , 97 , 465-480. · Zbl 1406.62021 · doi:10.1093/biomet/asq017
[13] Craven, P. and Wahba, G. (1979). Smoothing noisy data with spline functions: estimating the correct degree of smoothing by the method of generalized cross-validation., Numerische Mathematik , 31 , 377-403. · Zbl 0377.65007 · doi:10.1007/BF01404567
[14] Currie, I.D. and Durbán, M. (2002). Flexible smoothing with P-splines: a unified approach., Statistical Modelling , 4 , 333-349. · Zbl 1195.62072 · doi:10.1191/1471082x02st039ob
[15] Daubechies, I. (1988). Orthonormal bases of compactly supported wavelets., Communications on Pure and Applied Mathematics , 41 , 909-996. · Zbl 0644.42026 · doi:10.1002/cpa.3160410705
[16] Donoho, D.L. (1995). De-noising by soft-thresholding., IEEE Transactions on Information Theory , 41 , 613-627. · Zbl 0820.62002 · doi:10.1109/18.382009
[17] Donoho, D.L. and Johnstone, I.M. (1994). Ideal spatial adaptation by wavelet shrinkage., Biometrika , 81 , 425-456. · Zbl 0815.62019 · doi:10.1093/biomet/81.3.425
[18] Durbán, M., Harezlak, J., Wand, M.P. and Carroll, R.J. (2005). Simple fitting of subject-specific curves for longitudinal data., Statistics in Medicine , 24 , 1153-1167. · doi:10.1002/sim.1991
[19] Efron, B. (2004). The estimation of prediction error: covariance penalties and cross-validation (with discussion)., Journal of the American Statistical Association , 99 , 619-642. · Zbl 1117.62324 · doi:10.1198/016214504000000692
[20] Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression., The Annals of Statistics , 32 , 407-451. · Zbl 1091.62054 · doi:10.1214/009053604000000067
[21] Eilers, P.H.C. and Marx, B.D. (1996). Flexible smoothing with B-splines and penalties (with discussion)., Statistical Science , 11 , 89-121. · Zbl 0955.62562 · doi:10.1214/ss/1038425655
[22] Faes, C., Ormerod, J.T. and Wand, M.P. (2011). Variational Bayesian inference for parametric and nonparametric regression with missing data., Journal of the American Statistical Association , 106 , 959-971. · Zbl 1229.62028 · doi:10.1198/jasa.2011.tm10301
[23] Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties., Journal of the American Statistical Association , 96 , 1348-1360. · Zbl 1073.62547 · doi:10.1198/016214501753382273
[24] Fan, J. and Song, R. (2010). Sure independence screening in generalized linear models with NP-dimensionality., The Annals of Statistics , 38 , 3567-3604. · Zbl 1206.68157 · doi:10.1214/10-AOS798
[25] Fitzmaurice, G., Davidian, M., Verbeke, G. and Molenberghs, G. (Eds.) (2008)., Longitudinal Data Analysis: A Handbook of Modern Statistical Methods. Boca Raton, Florida: Chapman & Hall/CRC.
[26] Frank, I.E. and Friedman, J.H. (1993). A statistical view of some chemometrics regression tools., Technometrics , 35 , 109-135. · Zbl 0775.62288 · doi:10.2307/1269656
[27] Friedman, J., Hastie, T. and Tibshirani, R. (2009). glmnet 1.1: lasso and elastic-net regularized generalized linear models. R package.,
[28] Friedman, J., Hastie, T. and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent., Journal of Statistical Software , Volume 33, Issue 1, 1-22.
[29] Gelman, A. (2006). Prior distributions for variance parameters in hierarchical models., Bayesian Analysis , 1 , 515-533. · Zbl 1331.62139 · doi:10.1214/06-BA117A
[30] Gradshteyn, I.S. and Ryzhik, I.M. (1994)., Tables of Integrals, Series, and Products , 5th Edition. San Diego, California: Academic Press. · Zbl 0918.65002
[31] Green, P.J. and Silverman, B.W. (1994)., Nonparametric Regression and Generalized Linear Models. London: Chapman and Hall. · Zbl 0832.62032
[32] Griffin, J.E. and Brown, P.J. (2011). Bayesian hyper lassos with non-convex penalization., Australian and New Zealand Journal of Statistics , · Zbl 1335.62047
[33] Hart, J.P., McCurdy, M.R., Ezhil, M., Wei W., Khan, M., Luo, D., Munden, R.F., Johnson, V.E. and Guerrero, T.M. (2008). Radiation pneumonitis: correlation and toxicity with pulmonary metabolic radiation response., International Journal of Radiation Oncology, Biology, Physics , 4 , 967-971.
[34] Hastie, T. (1996). Pseudosplines., Journal of the Royal Statistical Society, Series B , 58 , 379-396. · Zbl 0853.62035
[35] Hastie, T. and Efron, B. (2007). lars 0.9. Least angle regression, lasso and forward stagewise regression. R package.,
[36] Hastie, T.J. and Tibshirani, R.J. (1990)., Generalized Additive Models . London: Chapman and Hall. · Zbl 0747.62061
[37] Hastie, T., Tibshirani, R. and Friedman, J. (2009)., The Elements of Statistical Learning , Second Edition. New York: Springer. · Zbl 1273.62005
[38] Hurvich, C. M., Simonoff, J. S. and Tsai, C. (1998). Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion., Journal of the Royal Statistical Society, Series B , 60 , 271-293. · Zbl 0909.62039 · doi:10.1111/1467-9868.00125
[39] Johnstone, I.M. and Silverman, B.W. (2005). Empirical Bayes selection of wavelet thresholds., The Annals of Statistics , 33 , 1700-1752. · Zbl 1078.62005 · doi:10.1214/009053605000000345
[40] Kerkyacharian, G. and Picard, D. (1992). Density estimation in Besov spaces., Statistics and Probability Letters , 13 , 15-24. · Zbl 0749.62026 · doi:10.1016/0167-7152(92)90231-S
[41] Ligges, U., Thomas, A., Spiegelhalter, D., Best, N. Lunn, D., Rice, K. and Sturtz, S. (2009). BRugs 0.5: OpenBUGS and its R/S-PLUS interface BRugs.,
[42] Marley, J.K. and Wand, M.P. (2010). Non-standard semiparametric regression via BRugs., Journal of Statistical Software , Volume 37, Issue 5, 1-30.
[43] Marron, J.S., Adak, S., Johnstone, I.M., Neumann, M.H. and Patil, P. (1998). Exact risk analysis of wavelet regression., Journal of Computational and Graphical Statistics , 7 , 278-309.
[44] Minka, T., Winn, J., Guiver, G. and Knowles, D. (2009). Infer.Net 2.4. Microsoft Research Cambridge, Cambridge, UK.,
[45] Morris, J.S. and Carroll, R.J. (2006). Wavelet-based functional mixed models., Journal of the Royal Statistical Society, Series B , 68 , 179-199. · Zbl 1110.62053 · doi:10.1111/j.1467-9868.2006.00539.x
[46] Morris, J.S., Vannucci, M., Brown, P.J. and Carroll, R.J. (2003). Wavelet-based nonparametric modeling of hierarchical functions in colon carcinogenesis., Journal of the American Statistical Association , 98 , 573-597. · Zbl 1040.62104 · doi:10.1198/016214503000000422
[47] Nason, G.P. (2008)., Wavelet Methods in Statistics with R. New York: Springer. · Zbl 1165.62033
[48] Nason, G.P. (2010). wavethresh 4.5. Wavelets statistics and transforms. R package.,
[49] Ormerod, J.T. and Wand, M.P. (2010). Explaining variational approximations., The American Statistician , 64 , 140-153. · Zbl 1200.65007 · doi:10.1198/tast.2010.09058
[50] Osborne, M.R., Presnell, B. and Turlach, B.A. (2000). On the LASSO and its dual., Journal of Computational and Graphical Statistics , 9 , 319-337.
[51] O’Sullivan, F. (1986). A statistical perspective on ill-posed inverse problems (with discussion)., Statistical Science , 1 , 505-527. · Zbl 0625.62110 · doi:10.1214/ss/1177013525
[52] Pearl, J. (1988)., Probabilistic Reasoning in Intelligent Systems . San Mateo, California: Morgan Kaufmann. · Zbl 0746.68089
[53] Pericchi, L.R. and Smith, A.F.M. (1992). Exact and approximate posterior moments for a normal location parameter., Journal of the Royal Statistical Society, Series B , 54 , 793-804. · Zbl 0775.62024
[54] R Development Core Team (2011). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0,
[55] Robert, C.P. and Casella, G. (2004)., Monte Carlo Statistical Methods , 2nd ed. New York: Springer. · Zbl 1096.62003
[56] Ruppert, D., Wand, M.P. and Carroll, R.J. (2003)., Semiparametric Regression . New York: Cambridge University Press. · Zbl 1038.62042
[57] Ruppert, D., Wand, M.P. and Carroll, R.J. (2009). Semiparametric regression during 2003-2007., Electronic Journal of Statistics , 3 , 1193-1256. · Zbl 1326.62094 · doi:10.1214/09-EJS525
[58] Spiegelhalter, D.J., Thomas, A., Best, N.G., Gilks, W.R. and Lunn, D. (2003). BUGS: Bayesian inference using Gibbs sampling. Medical Research Council Biostatistics Unit, Cambridge, UK., .
[59] Staudenmayer, J., Lake, E.E. and Wand, M.P. (2009). Robustness for general design mixed models using the, t -distribution. Statistical Modelling , 9 , 235-255. · doi:10.1177/1471082X0800900304
[60] Stein, C. (1981). Estimation of the mean of a multivariate normal distribution., The Annals of Statistics , 9 , 1135-1151. · Zbl 0476.62035 · doi:10.1214/aos/1176345632
[61] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso., Journal of the Royal Statistical Society, Series B , 58 , 267-288. · Zbl 0850.62538
[62] Vidakovic, B. (1999)., Statistical Modeling by Wavelets. New York: Wiley. · Zbl 0924.62032
[63] Wahba, G. (1990)., Spline Models for Observational Data , Philadelphia, Pennsylvania: Society for Industrial and Applied Mathematics. · Zbl 0813.62001
[64] Wainwright, M.J. and Jordan, M.I. (2008). Graphical models, exponential families, and variational inference., Foundation and Trends in Machine Learning , 1 , 1-305. · Zbl 1193.62107 · doi:10.1561/2200000001
[65] Wand, M.P. (2009). Semiparametric regression and graphical models., Australian and New Zealand Journal of Statistics , 51 , 9-41. · Zbl 1334.62012
[66] Wand, M.P. and Ormerod, J.T. (2008). On semiparametric regression with O’Sullivan penalized splines., Australian and New Zealand Journal of Statistics , 50 , 179-198. · Zbl 1146.62030 · doi:10.1111/j.1467-842X.2008.00507.x
[67] Wand, M.P., Ormerod, J.T., Padoan, S.A. and Frühwirth, R. (2011). Mean field variational Bayes for elaborate distributions., Bayesian Analysis , · Zbl 1330.62158 · doi:10.1214/11-BA631
[68] Wang, Y. (1998). Mixed effects smoothing spline analysis of variance., Journal of the Royal Statistical Society, Series B , 60 , 159-174. · Zbl 0909.62034 · doi:10.1111/1467-9868.00115
[69] Wang, S.S.J. and Wand, M.P. (2011). Using Infer.NET for statistical analyses., The American Statistician , 65 , 115-126. · Zbl 06244069 · doi:10.1198/tast.2011.10169
[70] Welham, S.J., Cullis, B.R., Kenward, M.G. and Thompson, R. (2007). A comparison of mixed model splines for curve fitting., Australian and New Zealand Journal of Statistics , 49 , 1-23. · Zbl 1117.62041 · doi:10.1111/j.1467-842X.2006.00454.x
[71] Wood, S.N. (2003). Thin-plate regression splines., Journal of the Royal Statistical Society, Series B , 65 , 95-114. · Zbl 1063.62059 · doi:10.1111/1467-9868.00374
[72] Wood, S.N. (2006)., Generalized Additive Models: An Introduction with R. Boca Raton, Florida: Chapman & Hall/CRC. · Zbl 1087.62082
[73] Wood, S.N. (2011). mgcv 1.7. GAMs with GCV/AIC/REML smoothness estimation and GAMMs by PQL. R package., · doi:10.1111/j.1467-9868.2010.00749.x
[74] Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty., The Annals of Statistics , 38 , 894-942. · Zbl 1183.62120 · doi:10.1214/09-AOS729
[75] Zhang, D., Lin, X., Raz, J. and Sowers, M. (1998). Semi-parametric stochastic mixed models for longitudinal data., Journal of the American Statistical Association , 93 , 710-719 · Zbl 0918.62039 · doi:10.2307/2670121
[76] Zhao, W. and Wu, R. (2008). Wavelet-based nonparametric functional mapping of longitudinal curves., Journal of the American Statistical Association , 103 , 714-725. · Zbl 1471.62562 · doi:10.1198/016214508000000373
[77] Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net., Journal of the Royal Statistical Society, Series B , 67 , 301-320. · Zbl 1069.62054 · doi:10.1111/j.1467-9868.2005.00503.x
[78] Zou, H., Hastie, T. and Tibshirani, R. (2007). On the “degrees of freedom” of the lasso., The Annals of Statistics , 5 , 2173-2192. · Zbl 1126.62061 · doi:10.1214/009053607000000127
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.