×

zbMATH — the first resource for mathematics

Online sparse identification for regression models. (English) Zbl 1447.93360
Summary: In this paper, we propose an online alternating minimization (OAM) algorithm to estimate the sparse coefficients of stochastic regression models from time-series data. We apply the alternating minimization (AM) directly to the penalty function of the variant of the least absolute shrinkage and selection operator (Lasso), which leads to convex subproblems, and thereby can be solved efficiently. Moreover, under certain mild assumptions, we derive a convergence analysis framework and establish the strong consistency for the OAM estimator. Numerical experiments demonstrate the effectiveness of the proposed algorithm.
MSC:
93E12 Identification in stochastic control theory
68W27 Online algorithms; streaming algorithms
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Tibshirani, R., Regression shrinkage and selection via the lasso, J. R. Stat. Soc. Ser. B Stat. Methodol., 58, 1, 267-288 (1996) · Zbl 0850.62538
[2] Wainwright, M. J., Sharp thresholds for High-Dimensional and noisy sparsity recovery using \(\ell_1\)-Constrained Quadratic Programming (Lasso), IEEE Trans. Inform. Theory, 55, 5, 2183-2202 (2009) · Zbl 1367.62220
[3] Zou, H.; Hastie, T., Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B Stat. Methodol., 67, 2, 301-320 (2005) · Zbl 1069.62054
[4] Candes, E.; Tao, T., The Dantzig selector: Statistical estimation when p is much larger than n, Ann. Statist., 35, 6, 2313-2351 (2007) · Zbl 1139.62019
[5] Tipping, M. E., Sparse Bayesian learning and the relevance vector machine, J. Mach. Learn. Res., 1, Jun, 211-244 (2001) · Zbl 0997.68109
[6] Yuan, Y.; Tang, X.; Zhou, W.; Pan, W.; Li, X.; Zhang, H.-T.; Ding, H.; Goncalves, J., Data driven discovery of cyber physical systems, Nature Commun., 10, 1, 1-9 (2019)
[7] Kou, P.; Gao, F.; Guan, X., Sparse online warped Gaussian process for wind power probabilistic forecasting, Appl. Energy, 108, 410-428 (2013)
[8] X. Gao, S.C. Hoi, Y. Zhang, J. Wan, J. Li, Soml: Sparse online metric learning with application to image retrieval, in: Twenty-Eighth AAAI Conference on Artificial Intelligence, 2014.
[9] Chen, H.-F.; Guo, L., Optimal adaptive control and consistent parameter estimates for ARMAX model with quadratic cost, SIAM J. Control Optim., 25, 4, 845-867 (1987) · Zbl 0632.93045
[10] Chen, H.-F.; Guo, L., Consistent estimation of the order of stochastic control systems, IEEE Trans. Autom. Control, 32, 6, 531-535 (1987) · Zbl 0615.93071
[11] Gu, Y.; Jin, J.; Mei, S., \( \ell_0\) Norm constraint LMS algorithm for sparse system identification, IEEE Signal Process. Lett., 16, 9, 774-777 (2009)
[12] Angelosante, D.; Bazerque, J. A.; Giannakis, G. B., Online adaptive estimation of sparse signals: Where RLS meets the \(\ell_1\)-norm, IEEE Trans. Signal Process., 58, 7, 3436-3447 (2010) · Zbl 1392.94073
[13] Babadi, B.; Kalouptsidis, N.; Tarokh, V., SPARLS: The sparse RLS algorithm, IEEE Trans. Signal Process., 58, 8, 4013-4025 (2010) · Zbl 1392.94080
[14] Eksioglu, E. M.; Tanc, A. K., RLS algorithm with convex regularization, IEEE Signal Process. Lett., 18, 8, 470-473 (2011)
[15] Eksioglu, E. M., Group sparse RLS algorithms, Internat. J. Adapt. Control Signal Process., 28, 12, 1398-1412 (2014) · Zbl 1334.93188
[16] Yu, L.; Wei, C.; Zheng, G., Group sparse LMS for multiple system identification, (2015 23rd European Signal Processing Conference. 2015 23rd European Signal Processing Conference, EUSIPCO (2015), IEEE), 1691-1695 · Zbl 1349.93119
[17] Yazdanpanah, H.; Diniz, P. S., Recursive least-squares algorithms for sparse system modeling, (2017 IEEE International Conference on Acoustics, Speech and Signal Processing. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP (2017), IEEE), 3879-3883
[18] Chen, H.-F.; Guo, L., Convergence rate of least-squares identification and adaptive control for stochastic systems, Internat. J. Control, 44, 5, 1459-1476 (1986) · Zbl 0604.93061
[19] Ljung, L., On positive real transfer functions and the convergence of some recursive schemes, IEEE Trans. Automat. Control, 22, 4, 539-551 (1977) · Zbl 0361.93063
[20] Chen, H.-f.; Guo, L., Identification and Stochastic Adaptive Control, Vol. 5 (1991), Springer Science & Business Media
[21] Slock, D. T., On the convergence behavior of the LMS and the normalized LMS algorithms, IEEE Trans. Signal Process., 41, 9, 2811-2825 (1993) · Zbl 0800.94093
[22] Douglas, S. C., A family of normalized LMS algorithms, IEEE Signal Process. Lett., 1, 3, 49-51 (1994)
[23] H. Wang, A. Banerjee, Online alternating direction method, in: Proceedings of the 29th International Coference on International Conference on Machine Learning, 2012, pp. 1699-1706.
[24] A. Choromanska, B. Cowen, S. Kumaravel, R. Luss, M. Rigotti, I. Rish, P. Diachille, V. Gurev, B. Kingsbury, R. Tejwani, et al. Beyond backprop: Online alternating minimization with auxiliary variables, in: International Conference on Machine Learning, 2019, pp. 1193-1202.
[25] Lai, T. L.; Wei, C. Z., Least squares estimates in stochastic regression models with applications to identification and control of dynamic systems, Ann. Statist., 10, 1, 154-166 (1982) · Zbl 0488.62071
[26] Rojas, C. R.; Tóth, R.; Hjalmarsson, H., Sparse estimation of polynomial and rational dynamical models, IEEE Trans. Automat. Control, 59, 11, 2962-2977 (2014) · Zbl 1360.93679
[27] Mao, K.; Billings, S., Algorithms for minimal model structure detection in nonlinear dynamic system identification, Internat. J. Control, 68, 2, 311-330 (1997) · Zbl 0889.93017
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.