×

Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks. (English) Zbl 1402.92030

Summary: We introduce a data-driven forecasting method for high-dimensional chaotic systems using long short-term memory (LSTM) recurrent neural networks. The proposed LSTM neural networks perform inference of high-dimensional dynamical systems in their reduced order space and are shown to be an effective set of nonlinear approximators of their attractor. We demonstrate the forecasting performance of the LSTM and compare it with Gaussian processes (GPs) in time series obtained from the Lorenz 96 system, the Kuramoto-Sivashinsky equation and a prototype climate model. The LSTM networks outperform the GPs in short-term forecasting accuracy in all applications considered. A hybrid architecture, extending the LSTM with a mean stochastic model (MSM-LSTM), is proposed to ensure convergence to the invariant measure. This novel hybrid method is fully data-driven and extends the forecasting capabilities of LSTM networks.

MSC:

92B20 Neural networks for/in biological studies, artificial life and related topics
62H30 Classification and discrimination; cluster analysis (statistical aspects)
86A32 Geostatistics

Software:

GNMT; Evolino; Adam
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Rowley, CW, Model reduction for fluids, using balanced proper orthogonal decomposition, Int. J. Bifurcation Chaos, 15, 997-1013, (2005) · Zbl 1140.76443 · doi:10.1142/S0218127405012429
[2] Williams, MO; Kevrekidis, IG; Rowley, CW, A data-driven approximation of the Koopman operator: extending dynamic mode decomposition, J. Nonlinear Sci., 25, 1307-1346, (2015) · Zbl 1329.65310 · doi:10.1007/s00332-015-9258-5
[3] Tu, JH; Rowley, CW; Luchtenburg, DM; Brunton, SL; Kutz, JN, On dynamic mode decomposition: theory and applications, J. Comput. Dyn., 1, 391-421, (2014) · Zbl 1346.37064 · doi:10.3934/jcd.2014.1.391
[4] Kutz, JN; Fu, X.; Brunton, SL, Multiresolution dynamic mode decomposition, SIAM. J. Appl. Dyn. Syst., 15, 713-735, (2016) · Zbl 1338.37121 · doi:10.1137/15M1023543
[5] Arbabi, H.; Mezic, I., Ergodic theory, dynamic mode decomposition and computation of spectral properties of the Koopman operator, SIAM. J. Appl. Dyn. Syst., 16, 2096-2126, (2017) · Zbl 1381.37096 · doi:10.1137/17M1125236
[6] Kerschen, G.; Golinval, JC; Vakakis, AF; Bergman, LA, The method of proper orthogonal decomposition for dynamical characterization and order reduction of mechanical systems: an overview, Nonlinear. Dyn., 41, 147-169, (2005) · Zbl 1103.70011 · doi:10.1007/s11071-005-2803-2
[7] Sapsis, TP; Majda, AJ, Statistically accurate low-order models for uncertainty quantification in turbulent dynamical systems, Proc. Natl Acad. Sci. USA, 110, 13 705-13 710, (2013) · Zbl 1292.62133 · doi:10.1073/pnas.1313065110
[8] Krischer, K.; Rico-Martínez, R.; Kevrekidis, IG; Rotermund, HH; Ertl, G.; Hudson, JL, Model identification of a spatiotemporally varying catalytic reaction, AIChE J., 39, 89-98, (1992) · doi:10.1002/aic.690390110
[9] Milano, M.; Koumoutsakos, P., Neural network modeling for near wall turbulent flow, J. Comput. Phys., 182, 1-26, (2002) · Zbl 1090.76535 · doi:10.1006/jcph.2002.7146
[10] Brunton, SL; Proctor, JL; Kutz, JN, Discovering governing equations from data by sparse identification of nonlinear dynamical systems, Proc. Natl Acad. Sci. USA, 113, 3932-3937, (2016) · Zbl 1355.94013 · doi:10.1073/pnas.1517384113
[11] Duriez, T.; Brunton, SL; Noack, BR, \(Machine learning control: taming nonlinear dynamics and turbulence\), (2016), Springer
[12] Majda, AJ; Lee, Y., Conceptual dynamical models for turbulence, Proc. Natl Acad. Sci. USA, 111, 6548-6553, (2014) · Zbl 1355.76025 · doi:10.1073/pnas.1404914111
[13] Schaeffer, H., Learning partial differential equations via data discovery and sparse optimization, Proc. R. Soc. A, 473, 20160446, (2017) · doi:10.1098/rspa.2016.0446
[14] Farazmand, M.; Sapsis, TP, Dynamical indicators for the prediction of bursting phenomena in high-dimensional systems, Phys. Rev. E, 94, 032212, (2016) · doi:10.1103/PhysRevE.94.032212
[15] Bongard, J.; Lipson, H., Automated reverse engineering of nonlinear dynamical systems, Proc. Natl Acad. Sci. USA, 104, 9943-9948, (2007) · Zbl 1155.37044 · doi:10.1073/pnas.0609476104
[16] Einicke, GA; White, LB, Robust extended Kalman filtering, IEEE Trans. Signal Process., 47, 2596-2599, (1999) · Zbl 1006.93588 · doi:10.1109/78.782219
[17] Julier, SJ; Uhlmann, JK, A new extension of the Kalman filter to nonlinear systems, Proc. SPIE, 3068, 182-193, (1997) · doi:10.1117/12.280797
[18] Lee, Y.; Majda, AJ, State estimation and prediction using clustered particle filters, Proc. Natl Acad. Sci. USA, 113, 14 609-14 614, (2016) · doi:10.1073/pnas.1617398113
[19] Comeau, D.; Zhao, Z.; Giannakis, D.; Majda, AJ, Data-driven prediction strategies for low-frequency patterns of North Pacific climate variability, Clim. Dyn., 48, 1855-1872, (2017) · doi:10.1007/s00382-016-3177-5
[20] Tatsis, K.; Dertimanis, V.; Abdallah, I.; Chatzi, E., A substructure approach for fatigue assessment on wind turbine support structures using output-only measurements, Procedia Engineering, 199, 1044-1049, (2017) · doi:10.1016/j.proeng.2017.09.285
[21] Quade, M.; Abel, M.; Shafi, K.; Niven, RK; Noack, BR, Prediction of dynamical systems by symbolic regression, Phys. Rev. E., 94, 012214, (2016) · doi:10.1103/PhysRevE.94.012214
[22] Mirmomeni, M.; Lucas, C.; Moshiri, B.; Araabi, NB, Introducing adaptive neurofuzzy modeling with online learning method for prediction of time-varying solar and geomagnetic activity indices, Expert Syst. Appl., 37, 8267-8277, (2010) · doi:10.1016/j.eswa.2010.05.059
[23] Gholipour, A.; Lucas, C.; Araabi, NB; Mirmomeni, M.; Shafiee, M., Extracting the main patterns of natural time series for long-term neurofuzzy prediction, Neural Comput. Appl., 16, 383-393, (2007) · doi:10.1007/s00521-006-0062-x
[24] Mirmomeni, M.; Lucas, C.; Araabi, NB; Moshiri, B.; Bidar, MR, Recursive spectral analysis of natural time series based on eigenvector matrix perturbation for online applications, IET Signal Process., 5, 512-526, (2011) · doi:10.1049/iet-spr.2009.0278
[25] Mirmomeni, M.; Lucas, C.; Araabi, NB; Moshiri, B.; Bidar, MR, Online multi-step ahead prediction of time-varying solar and geomagnetic activity indices via adaptive neurofuzzy modeling and recursive spectral analysis, Sol. Phys., 272, 189-213, (2011) · doi:10.1007/s11207-011-9810-x
[26] Marques, CAF; Ferreira, JA; Rocha, A.; Castanheira, JM; Melo-Goncalves, P.; Vaz, N.; Dias, JM, Singular spectrum analysis and forecasting of hydrological time series, Phys. Chem. Earth, Parts A/B/C, 31, 1172-1179, (2006) · doi:10.1016/j.pce.2006.02.061
[27] Abdollahzade, M.; Miranian, A.; Hassani, H.; Iranmanesh, H., A new hybrid enhanced local linear neuro-fuzzy model based on the optimized singular spectrum analysis and its application for nonlinear and chaotic time series forecasting, Inf. Sci. (Ny), 295, 107-125, (2015) · doi:10.1016/j.ins.2014.09.002
[28] Ye, L.; Liu, P., Combined model based on EMD-SVM for short-term wind power prediction, Commun. Nonlinear Science Numer. Simul., 31, 102-108, (2011)
[29] Cousins, W.; Sapsis, TP, Quantification and prediction of extreme events in a one-dimensional nonlinear dispersive wave model, Phys. D, 280-281, 48-58, (2014) · Zbl 1349.35050 · doi:10.1016/j.physd.2014.04.012
[30] Cousins, W.; Sapsis, TP, Reduced order precursors of rare events in unidirectional nonlinear water waves, J. Fluid Mech., 790, 368-388, (2016) · Zbl 1382.76030 · doi:10.1017/jfm.2016.13.368
[31] Lorenz, EN, Atmospheric predictability as revealed by naturally occurring analogues, J. Atmos. Sci., 26, 636-646, (1969) · doi:10.1175/1520-0469(1969)26<636:APARBN>2.0.CO;2
[32] Xavier, PK; Goswami, BN, An analog method for real-time forecasting of summer monsoon subseasonal variability, Mon. Weather Rev., 135, 4149-4160, (2007) · doi:10.1175/2007MWR1854.1
[33] Zhao, Z.; Giannakis, D., Analog forecasting with dynamics-adapted kernels, Nonlinearity, 29, 2888-2939, (2016) · Zbl 1365.37062 · doi:10.1088/0951-7715/29/9/2888
[34] Chiavazzo, E.; Gear, CW; Dsilva, CJ; Rabin, N.; Kevrekidis, IG, Reduced models in chemical kinetics via nonlinear data-mining, Processes, 2, 112-140, (2014) · doi:10.3390/pr2010112
[35] Wan, ZY; Sapsis, TP, Reduced-space Gaussian Process Regression for data-driven probabilistic forecast of chaotic dynamical systems, Phys. D: Nonlinear Phenom., 345, 45-55, (2017) · Zbl 1380.37111 · doi:10.1016/j.physd.2016.12.005
[36] Rasmussen, CE; Williams, CKI, \(Gaussian processes for machine learning\), (2006), The MIT Press
[37] Hochreiter, S.; Schmidhuber, J., Long short-term memory, Neural. Comput., 9, 1735-1780, (1997) · doi:10.1162/neco.1997.9.8.1735
[38] Gers, FA; Eck, D.; Schmidhuber, J., Applying LSTM to time series predictable through time-window approaches. In \(Proc. Int. Conf. on Artificial Neural Networks, ICANN’01 Vienna, 21-25 August\) (eds G Dorffner, H Bischof, K Hornik), pp. 669-676. Lecture Notes in Computer Science, vol. 2130. New York, NY: Springer, (2001)
[39] Rico-Martínez, R.; Krischer, K.; Kevrekidis, IG; Kube, MC; Hudson, JL, Discrete- versus continuous-time nonlinear signal processing of Cu electrodissolution data, Chem. Eng. Commun., 118, 25-48, (1992) · doi:10.1080/00986449208936084
[40] Jaeger, M.; Haas, H., Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication, Science, 304, 78-80, (2004) · doi:10.1126/science.1091277
[41] Chatzis, SP; Demiris, Y., Echo state Gaussian process, IEEE Trans. Neural Netw., 22, 1435-1445, (2011) · doi:10.1109/TNN.2011.2162109
[42] Broomhead, DS; Lowe, D., Multivariable functional interpolation and adaptive networks, Complex Syst., 2, 321-355, (1988) · Zbl 0657.68085 · doi:10.4236/jbise.2013.65A003
[43] Kim, KB; Park, JB; Choi, YH; Chen, G., Control of chaotic dynamical systems using radial basis function network approximators, Inf. Sci., 130, 165-183, (2000) · Zbl 0981.93512 · doi:10.1016/S0020-0255(00)00074-8
[44] Pathak, J.; Hunt, BR; Girvan, M.; Lu, Z.; Ott, E., Model-free prediction of large spatiotemporally chaotic systems from data: a reservoir computing approach, Phys. Rev. Lett., 120, 024102, (2018) · doi:10.1103/PhysRevLett.120.024102
[45] Pathak, J.; Lu, Z.; Hunt, BR; Girvan, M.; Ott, E., Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data, Chaos., 27, 121102, (2017) · Zbl 1390.37138 · doi:10.1063/1.5010300
[46] Takens, F., Detecting strange attractors in turbulence. In \(Dynamical Systems and Turbulence, Warwick, UK, 1980\), pp. 366-381. Lecture Notes in Mathematics, vol. 898. Berlin, Germany: Springer, (1981)
[47] Hochreiter, J., Untersuchungen zu dynamischen neuronalen Netzen. Master thesis, Institut fur Informatik, Technische Universitat, Munchen, (1991)
[48] Bengio, Y.; Simard, P.; Frasconi, P., Long short-term memory, IEEE Trans. Neural Netw., 5, 157-166, (1994) · doi:10.1109/72.279181
[49] Wierstra, D.; Schmidhuber, J.; Gomez, FJ, Evolino: hybrid neuroevolution/optimal linear search for sequence learning. \(Proc. 19th IJCAI\), 853-858, (2005)
[50] Graves, A.; Mohamed, AR; Hinton, G., Speech recognition with deep recurrent neural networks. In \(Proc. Acoustic, Speech and Signal Processing, ICASSP, Vancouver, Canada, 26-31 May\), pp. 6645-6649. Piscataway, NJ: IEEE, (2013)
[51] Graves, A.; Fernández, S.; Liwicki, M.; Bunke, H.; Schmidhuber, J., Unconstrained online handwriting recognition with recurrent neural networks. In \(Proc. 20th NIPS, Vancouver, Canada, 3-6 December\), pp. 577-584. Curran Associates, (2007)
[52] Wu, Y.; Schuster, M.; Chen, Z.; Le, QV; Norouzi, M.; Macherey, W.; Krikun, M.; Cao, Y.; Gao, Q., Google’s neural machine translation system: bridging the gap between human and machine translation. (http://arxiv.org/abs/1609.08144), (2016)
[53] Kingma, DP; Ba, J., Adam: a method for stochastic optimization. (http://arxiv.org/abs/1412.6980), (2017)
[54] Glorot, X.; Bengio, Y., Understanding the difficulty of training deep feedforward neural networks. \(Proc. 13th AISTATS\)114, 249-256, (2010)
[55] Majda, A.; Harlim, J., \(Filtering complex turbulent systems\), (2012), Cambridge University Press · Zbl 1250.93002
[56] Majda, A.; Abramov, RV; Grote, MJ, Centre de Recherches Mathematiques Monograph Series, vol. 25. Providence, RI: American Mathematical Society, \(Information theory and stochastics for multiscale nonlinear systems\), (2005)
[57] Allgaier, NA; Harris, KD; Danforth, CM, Empirical correction of a toy climate model, Phys. Rev. E, 85, 026201, (2012) · doi:10.1103/PhysRevE.85.026201
[58] Basnarkov, L.; Kocarev, L., Forecast improvement in Lorenz96 system, Nonlinear Process. Geophys., 19, 569-575, (2012) · doi:10.5194/npg-19-569-2012
[59] Lorenz, NE, Predictability: a problem partly solved. In \(Proc. Seminar held at ECMWF on Predictability. Reading, UK, 4-8 September\), pp. 1-18, (1996)
[60] Crommelin, DT; Majda, AJ, Strategies for model reduction: comparing different optimal bases, J. Atmos. Sci., 61, 2206-2217, (2004) · doi:10.1175/1520-0469(2004)061<2206:SFMRCD>2.0.CO;2
[61] Kuramoto, Y.; Tsuzuki, T., Persistent propagation of concentration waves in dissipative media far from thermal equilibrium, Prog. Theor. Phys., 55, 356-369, (1976) · doi:10.1143/PTP.55.356
[62] Kuramoto, Y., Diffusion-induced chaos in reaction systems, Prog. Theor. Phys. Suppl., 64, 346-367, (1978) · doi:10.1143/PTPS.64.346
[63] Sivashinsky, G., Nonlinear analysis of hydrodynamic instability in laminar flames-I. Derivation of basic equations, Acta. Astronaut., 4, 1177-1206, (1977) · Zbl 0427.76047 · doi:10.1016/0094-5765(77)90096-0
[64] Sivashinsky, G.; Michelson, DM, On irregular wavy flow of a liquid film down a vertical plane, Prog. Theor. Phys., 63, 2112-2114, (1980) · doi:10.1143/PTP.63.2112
[65] Blonigan, PJ; Wang, Q., Least squares shadowing sensitivity analysis of a modified Kuramoto-Sivashinsky equation, Chaos Solitons Fractals, 64, 16-25, (2014) · Zbl 1348.35025 · doi:10.1016/j.chaos.2014.03.005
[66] Kevrekidis, IG; Nicolaenko, B.; Scovel, JC, Back in the saddle again: a computer assisted study of the kuramoto-sivashinsky equation, SIAM J. Appl. Math., 50, 760-790, (1990) · Zbl 0722.35011 · doi:10.1137/0150045
[67] Selten, FM, An efficient description of the dynamics of barotropic flow, J. Atmos. Sci., 52, 915-936, (1995) · doi:10.1175/1520-0469(1995)052<0915:AEDOTD>2.0.CO;2
[68] Thompson, DWJ; Wallace, JM, Annular modes in the extratropical circulation: Part I: Month-to-month variability, J. Clim., 13, 1000-1016, (2000) · doi:10.1175/1520-0442(2000)013<1000:AMITEC>2.0.CO;2
[69] Mo, KC; Livezey, RE, Tropical-extratropical geopotential height teleconnections during the northern hemisphere winter, Mon. Weather Rev., 114, 2488-2512, (1986) · doi:10.1175/1520-0493(1986)114<2488:TEGHTD>2.0.CO;2
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.