×

Predicting critical transitions in multiscale dynamical systems using reservoir computing. (English) Zbl 1457.37099

Summary: We study the problem of predicting rare critical transition events for a class of slow-fast nonlinear dynamical systems. The state of the system of interest is described by a slow process, whereas a faster process drives its evolution and induces critical transitions. By taking advantage of recent advances in reservoir computing, we present a data-driven method to predict the future evolution of the state. We show that our method is capable of predicting a critical transition event at least several numerical time steps in advance. We demonstrate the success as well as the limitations of our method using numerical experiments on three examples of systems, ranging from low dimensional to high dimensional. We discuss the mathematical and broader implications of our results.
©2020 American Institute of Physics

MSC:

37M05 Simulation of dynamical systems
70K70 Systems with slow and fast motions for nonlinear problems in mechanics
70K55 Transition to stochasticity (chaotic behavior) for nonlinear problems in mechanics
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Agarwal, S.; Wettlaufer, J. S., Maximal stochastic transport in the Lorenz equations, Phys. Lett. A, 380, 142-146 (2016) · Zbl 1377.76015 · doi:10.1016/j.physleta.2015.09.046
[2] Arnold, L., “Hasselmann’s program revisited: The analysis of stochasticity in deterministic climate models,” in Stochastic Climate Models (Springer, 2001), pp. 141-157. · Zbl 1015.34037
[3] Ashwin, P.; Wieczorek, S.; Vitolo, R.; Cox, P., Tipping points in open systems: Bifurcation, noise-induced and rate-dependent examples in the climate system, Philos. Trans. R. Soc. A: Math. Phys. Eng. Sci., 370, 1166-1184 (2012) · doi:10.1098/rsta.2011.0306
[4] Baker, N., Alexander, F., Bremer, T., Hagberg, A., Kevrekidis, Y., Najm, H., Parashar, M., Patra, A., Sethian, J., Wild, S.et al., “Workshop report on basic research needs for scientific machine learning: Core technologies for artificial intelligence,” Technical Report, U.S. DOE Office of Science (SC), Washington, DC, 2019.
[5] Beck, C., Brownian motion from deterministic dynamics, Physica A, 169, 324-336 (1990) · doi:10.1016/0378-4371(90)90173-P
[6] Bengio, Y., Learning deep architectures for AI, Found. Trends Mach. Learn., 2, 1-127 (2009) · Zbl 1192.68503 · doi:10.1561/2200000006
[7] Benzi, R.; Parisi, G.; Sutera, A.; Vulpiani, A., Stochastic resonance in climatic change, Tellus, 34, 10-16 (1982) · doi:10.3402/tellusa.v34i1.10782
[8] Benzi, R.; Sutera, A.; Vulpiani, A., The mechanism of stochastic resonance, J. Phys. A: Math. Gen., 14, L453 (1981) · doi:10.1088/0305-4470/14/11/006
[9] Berglund, N.; Gentz, B., Noise-Induced Phenomena in Slow-Fast Dynamical Systems: A Sample-Paths Approach (2006), Springer Science & Business Media · Zbl 1098.37049
[10] Berner, J.; Achatz, U.; Batté, L.; Bengtsson, L.; Cámara, A. D. L.; Christensen, H. M.; Colangeli, M.; Coleman, D. R.; Crommelin, D.; Dolaptchiev, S. I., Stochastic parameterization: Toward a new view of weather and climate models, Bull. Am. Meteorol. Soc., 98, 565-588 (2017) · doi:10.1175/BAMS-D-15-00268.1
[11] Bo, S.; Schmidt, F.; Eichhorn, R.; Volpe, G., Measurement of anomalous diffusion using recurrent neural networks, Phys. Rev. E, 100, 010102 (2019) · doi:10.1103/PhysRevE.100.010102
[12] Brajard, J.; Carrassi, A.; Bocquet, M.; Bertino, L., Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: A case study with the Lorenz 96 model, Geosci. Model Dev. Discuss., 2019, 1-21 · doi:10.5194/gmd-2019-136
[13] Brunton, S. L.; Kutz, J. N., Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control (2019), Cambridge University Press · Zbl 1407.68002
[14] Carleo, G.; Cirac, I.; Cranmer, K.; Daudet, L.; Schuld, M.; Tishby, N.; Vogt-Maranto, L.; Zdeborová, L., Machine learning and the physical sciences, Rev. Mod. Phys., 91, 045002 (2019) · doi:10.1103/RevModPhys.91.045002
[15] Chattopadhyay, A.; Hassanzadeh, P.; Subramanian, D., Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: Reservoir computing, artificial neural network, and long short-term memory network, Nonlinear Process. Geophys., 27, 373-389 (2020) · doi:10.5194/npg-27-373-2020
[16] Chevyrev, I., Friz, P. K., Korepanov, A., Melbourne, I., and Zhang, H., “Multiscale systems, homogenization, and rough paths,” in International Conference in Honor of the 75th Birthday of SRS Varadhan (Springer, 2016), pp. 17-48. · Zbl 1443.60060
[17] Cohen, A. B.; Ravoori, B.; Murphy, T. E.; Roy, R., Using synchronization for prediction of high-dimensional chaotic dynamics, Phys. Rev. Lett., 101, 154102 (2008) · doi:10.1103/PhysRevLett.101.154102
[18] Cuchiero, C.; Gonon, L.; Grigoryeva, L.; Ortega, J. P.; Teichmann, J.
[19] E., W.; Vanden-Eijnden, E., Towards a theory of transition paths, J. Stat. Phys., 123, 503 (2006) · Zbl 1101.82016 · doi:10.1007/s10955-005-9003-9
[20] Eisenman, I.; Wettlaufer, J. S., Nonlinear threshold behavior during the loss of Arctic sea ice, Proc. Natl. Acad. Sci. U.S.A., 106, 28-32 (2009) · Zbl 1355.86002 · doi:10.1073/pnas.0806887106
[21] Forgoston, E.; Moore, R. O., A primer on noise-induced transitions in applied dynamical systems, SIAM Rev., 60, 969-1009 (2018) · Zbl 1408.37089 · doi:10.1137/17M1142028
[22] Gallicchio, C.; Micheli, A.
[23] Gallicchio, C. and Scardapane, S., “Deep randomized neural networks,” in Recent Trends in Learning from Data (Springer, 2020), pp. 43-68.
[24] Gammaitoni, L.; Hänggi, P.; Jung, P.; Marchesoni, F., Stochastic resonance, Rev. Mod. Phys., 70, 223 (1998) · doi:10.1103/RevModPhys.70.223
[25] García, L. P.; Pérez, J. D.; Volpe, G.; Arzola, A. V.; Volpe, G., High-performance reconstruction of microscopic force fields from Brownian trajectories, Nat. Commun., 9, 5166 (2018) · doi:10.1038/s41467-018-07437-x
[26] Gaspard, P.; Briggs, M.; Francis, M.; Sengers, J.; Gammon, R.; Dorfman, J. R.; Calabrese, R., Experimental evidence for microscopic chaos, Nature, 394, 865 (1998) · doi:10.1038/29721
[27] Giorgini, L.; Lim, S.; Moon, W.; Wettlaufer, J., Precursors to rare events in stochastic resonance, Europhys. Lett., 129, 40003 (2020) · doi:10.1209/0295-5075/129/40003
[28] Givon, D.; Kupferman, R.; Stuart, A., Extracting macroscopic dynamics: Model problems and algorithms, Nonlinearity, 17, R55 (2004) · Zbl 1073.82038 · doi:10.1088/0951-7715/17/6/R01
[29] Gonon, L.; Grigoryeva, L.; Ortega, J.-P.
[30] Gonon, L.; Grigoryeva, L.; Ortega, J.-P.
[31] Goodfellow, I.; Bengio, Y.; Courville, A., Deep Learning (2016), MIT Press · Zbl 1373.68009
[32] Gottwald, G. A.; Melbourne, I., Homogenization for deterministic maps and multiplicative noise, Proc. R. Soc. A: Math. Phys. Eng. Sci., 469, 20130201 (2013) · Zbl 1371.34084 · doi:10.1098/rspa.2013.0201
[33] Grigoryeva, L.; Ortega, J.-P., Echo state networks are universal, Neural Netw., 108, 495-508 (2018) · Zbl 1434.68409 · doi:10.1016/j.neunet.2018.08.025
[34] Hart, A.; Hook, J.; Dawes, J., Embedding and approximation theorems for echo state networks, Neural Netw., 128, 234-247 (2020) · Zbl 1468.68098 · doi:10.1016/j.neunet.2020.05.013
[35] Hartmann, C.; Banisch, R.; Sarich, M.; Badowski, T.; Schütte, C., Characterization of rare events in molecular dynamics, Entropy, 16, 350-376 (2014) · doi:10.3390/e16010350
[36] Hasselmann, K., Stochastic climate models part I. Theory, Tellus, 28, 473-485 (1976) · doi:10.3402/tellusa.v28i6.11316
[37] Ilies, I., Jaeger, H., Kosuchinas, O., Rincon, M., Sakenas, V., and Vaskevicius, N., “Stepping forward through echoes of the past: Forecasting with echo state networks” (2007), Vol. 53, p. 76, Short report on the winning entry to the NN3 financial forecasting competition, see http://www.neural-forecasting-competition.com/downloads/NN3/methods/27-NN3HerbertJaegerreport.pdf.
[38] Jaeger, H., “Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the “echo state network” approach,” Vol. 5, GMD-Forschungszentrum Informationstechnik Bonn, 2002.
[39] Jaeger, H.; Haas, H., Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication, Science, 304, 78-80 (2004) · doi:10.1126/science.1091277
[40] Kaplan, D. T.; Glass, L., Coarse-grained embeddings of time series: Random walks, Gaussian random processes, and deterministic chaos, Physica D, 64, 431-454 (1993) · Zbl 0773.58016 · doi:10.1016/0167-2789(93)90054-5
[41] Kelly, D.; Melbourne, I., Deterministic homogenization for fast-slow systems with chaotic noise, J. Funct. Anal., 272, 4063-4102 (2017) · Zbl 1373.60101 · doi:10.1016/j.jfa.2017.01.015
[42] Kelly, D.; Melbourne, I., Smooth approximation of stochastic differential equations, Ann. Probab., 44, 479-520 (2016) · Zbl 1372.60082 · doi:10.1214/14-AOP979
[43] Kuehn, C., A mathematical framework for critical transitions: Bifurcations, fast-slow systems and stochastic dynamics, Physica D, 240, 1020-1035 (2011) · Zbl 1225.35242 · doi:10.1016/j.physd.2011.02.012
[44] LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P., Gradient-based learning applied to document recognition, Proc. IEEE, 86, 2278-2324 (1998) · doi:10.1109/5.726791
[45] Lim, S. H.
[46] Lorenz, E. N., Deterministic nonperiodic flow, J. Atmos. Sci., 20, 130-141 (1963) · Zbl 1417.37129 · doi:10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2
[47] Lorenz, E. N., “Predictability: A problem partly solved,” in Seminar Proceedings on Predictability (ECMWF, 1995), Vol. 1.
[48] Lu, Z.; Hunt, B. R.; Ott, E., Attractor reconstruction by machine learning, Chaos, 28, 061104 (2018) · doi:10.1063/1.5039508
[49] Lukoševičius, M., “A practical guide to applying echo state networks,” in Neural Networks: Tricks of the Trade (Springer, 2012), pp. 659-686.
[50] Lukoševičius, M.; Jaeger, H., Reservoir computing approaches to recurrent neural network training, Comput. Sci. Rev., 3, 127-149 (2009) · Zbl 1302.68235 · doi:10.1016/j.cosrev.2009.03.005
[51] Lukoševičius, M. and Uselis, A., “Efficient cross-validation of echo state networks,” in Artificial Neural Networks and Machine Learning—ICANN 2019: Workshop and Special Sessions, edited by I. V. Tetko, V. Kůrková, P. Karpov, and F. Theis (Springer International Publishing, Cham, 2019), pp. 121-133.
[52] Lusch, B.; Kutz, J. N.; Brunton, S. L., Deep learning for universal linear embeddings of nonlinear dynamics, Nat. Commun., 9, 4950 (2018) · doi:10.1038/s41467-018-07210-0
[53] Mackey, M. C.; Tyran-Kamińska, M., Deterministic Brownian motion: The effects of perturbing a dynamical system by a chaotic semi-dynamical system, Phys. Rep., 422, 167-222 (2006) · doi:10.1016/j.physrep.2005.09.002
[54] Manjunath, G.; Jaeger, H., Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks, Neural Comput., 25, 671-696 (2013) · Zbl 1269.92006 · doi:10.1162/NECO_a_00411
[55] Mehta, P.; Bukov, M.; Wang, C.-H.; Day, A. G.; Richardson, C.; Fisher, C. K.; Schwab, D. J., A high-bias, low-variance introduction to machine learning for physicists, Phys. Rep., 810, 1-124 (2019) · doi:10.1016/j.physrep.2019.03.001
[56] Melbourne, I.; Stuart, A., A note on diffusion limits of chaotic skew-product flows, Nonlinearity, 24, 1361 (2011) · Zbl 1220.37009 · doi:10.1088/0951-7715/24/4/018
[57] Mitchell, L.; Gottwald, G. A., Data assimilation in slow-fast systems using homogenized climate models, J. Atmos. Sci., 69, 1359-1377 (2012) · doi:10.1175/JAS-D-11-0145.1
[58] Doing so will usually, but not always, ensure that the resulting network satisfies the echo state property.\({}^{38}\)
[59] Pascanu, R.; Gulcehre, C.; Cho, K.; Bengio, Y.
[60] Pathak, J.; Hunt, B.; Girvan, M.; Lu, Z.; Ott, E., Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach, Phys. Rev. Lett., 120, 024102 (2018) · doi:10.1103/PhysRevLett.120.024102
[61] Pathak, J.; Lu, Z.; Hunt, B. R.; Girvan, M.; Ott, E., Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data, Chaos, 27, 121102 (2017) · Zbl 1390.37138 · doi:10.1063/1.5010300
[62] Pathak, J.; Wikner, A.; Fussell, R.; Chandra, S.; Hunt, B. R.; Girvan, M.; Ott, E., Hybrid forecasting of chaotic processes: Using machine learning in conjunction with a knowledge-based model, Chaos, 28, 041101 (2018) · doi:10.1063/1.5028373
[63] Pavliotis, G. and Stuart, A., Multiscale Methods, Texts in Applied Mathematics Vol. 53 (Springer, New York, 2008). · Zbl 1160.35006
[64] Pecora, L. M.; Carroll, T. L., Synchronization of chaotic systems, Chaos, 25, 097611 (2015) · Zbl 1374.34002 · doi:10.1063/1.4917383
[65] Raissi, M.; Perdikaris, P.; Karniadakis, G. E.
[66] Riley, P., Three pitfalls to avoid in machine learning, Nature, 572, 27-29 (2019) · doi:10.1038/d41586-019-02307-y
[67] Rödenbeck, C., Beck, C., and Kantz, H., “Dynamical systems with time scale separation: Averaging, stochastic modelling, and central limit theorems,” in Stochastic Climate Models (Springer, 2001), pp. 189-209. · Zbl 0990.34042
[68] Rolnick, D.; Donti, P. L.; Kaack, L. H.; Kochanski, K.; Lacoste, A.; Sankaran, K.; Slavin Ross, A.; Milojevic-Dupont, N.; Jaques, N.; Waldman-Brown, A.
[69] Romanczuk, P.; Bär, M.; Ebeling, W.; Lindner, B.; Schimansky-Geier, L., Active Brownian particles, Eur. Phys. J. Spec. Top., 202, 1-162 (2012) · doi:10.1140/epjst/e2012-01529-y
[70] Scardapane, S.; Wang, D., Randomness in neural networks: An overview, Wiley Interdiscip. Rev. Data Min. Knowl. Discov., 7, e1200 (2017) · doi:10.1002/widm.1200
[71] Scheffer, M.; Bascompte, J.; Brock, W. A.; Brovkin, V.; Carpenter, S. R.; Dakos, V.; Held, H.; Van Nes, E. H.; Rietkerk, M.; Sugihara, G., Early-warning signals for critical transitions, Nature, 461, 53-59 (2009) · doi:10.1038/nature08227
[72] Scher, S., Toward data-driven weather and climate forecasting: Approximating a simple general circulation model with deep learning, Geophys. Res. Lett., 45, 12616-12622 (2018) · doi:10.1029/2018GL080704
[73] Sherstinsky, A., Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network, Physica D, 404, 132306 (2020) · Zbl 1508.68298 · doi:10.1016/j.physd.2019.132306
[74] Stone, H., Calibrating rough volatility models: A convolutional neural network approach, Quant. Finance, 20, 379-392 (2020) · Zbl 1466.91318 · doi:10.1080/14697688.2019.1654126
[75] Tino, P., Dynamical systems as temporal feature spaces, J. Mach. Learn. Res., 21, 1-42 (2020) · Zbl 1498.68268
[76] Verzelli, P.; Alippi, C.; Livi, L.
[77] Verzelli, P.; Alippi, C.; Livi, L.; Tino, P.
[78] Vlachas, P. R.; Byeon, W.; Wan, Z. Y.; Sapsis, T. P.; Koumoutsakos, P., Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks, Proc. R. Soc. A: Math. Phys. Eng. Sci., 474, 20170844 (2018) · Zbl 1402.92030 · doi:10.1098/rspa.2017.0844
[79] Vlachas, P. R.; Pathak, J.; Hunt, B. R.; Sapsis, T. P.; Girvan, M.; Ott, E.; Koumoutsakos, P., Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics, Neural Netw., 126, 191-217 (2020) · doi:10.1016/j.neunet.2020.02.016
[80] Weng, T.; Yang, H.; Gu, C.; Zhang, J.; Small, M., Synchronization of chaotic systems and their machine-learning models, Phys. Rev. E, 99, 042203 (2019) · doi:10.1103/PhysRevE.99.042203
[81] Yehudai, G. and Shamir, O., “On the power and limitations of random features for understanding neural networks,” in Advances in Neural Information Processing Systems (Curran Associates, Inc., 2019), Vol. 32, pp. 6594-6604.
[82] Yperman, J.; Becker, T.
[83] Zimmermann, R. S.; Parlitz, U., Observing spatio-temporal dynamics of excitable media using reservoir computing, Chaos, 28, 043118 (2018) · doi:10.1063/1.5022276
[84] Tanaka, G.; Yamane, T.; H{\'e}roux, J. B.; Nakane, R.; Kanazawa, N.; Takeda, S.; Numata, H.; Nakano, D.; Hirose, A., Recent advances in physical reservoir computing: A review, Neural Networks, 115, 100-123 (2019) · doi:10.1016/j.neunet.2019.03.005
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.