×

Echo state networks with filter neurons and a delay&sum readout. (English) Zbl 1396.68094

Summary: Echo state networks (ESNs) are a novel approach to recurrent neural network training with the advantage of a very simple and linear learning algorithm. It has been demonstrated that ESNs outperform other methods on a number of benchmark tasks. Although the approach is appealing, there are still some inherent limitations in the original formulation.
Here we suggest two enhancements of this network model. First, the previously proposed idea of filters in neurons is extended to arbitrary infinite impulse response (IIR) filter neurons. This enables such networks to learn multiple attractors and signals at different timescales, which is especially important for modeling real-world time series. Second, a delay&sum readout is introduced, which adds trainable delays in the synaptic connections of output neurons and therefore vastly improves the memory capacity of echo state networks.
It is shown in commonly used benchmark tasks and real-world examples, that this new structure is able to significantly outperform standard ESNs and other state-of-the-art models for nonlinear dynamical system modeling.

MSC:

68T05 Learning and adaptive systems in artificial intelligence

Software:

Evolino
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Atiya, A.F.; Parlos, A.G., New results on recurrent network training: unifying the algorithms and accelerating convergence, Ieee-nn, 11, 3, 697, (2000)
[2] Ausserlechner, H. (2006). Untersuchungen von dropout-concealment-algorithmen. Institute for Electronic Music and Acoustics, Graz, Austria. Diplomarbeit; Ausserlechner, H. (2006). Untersuchungen von dropout-concealment-algorithmen. Institute for Electronic Music and Acoustics, Graz, Austria. Diplomarbeit
[3] Azaria, M.; Hertz, D., Time delay estimation by generalized cross correlation methods, Acoustics, speech, and signal processing; IEEE transactions on signal processing, 32, 285-380, (1986)
[4] Bringuier, V.; Chavane, F.; Glaeser, L.; Frégnac, Y., Horizontal propagation of visual activity in the synaptic integration field of area 17 neurons, Science, 283, 695-699, (1999)
[5] Bristow-Johnson, R. (2008). Cookbook formulae for audio eq biquad filter coefficients. Online resource, accessed 14.4.2008; Bristow-Johnson, R. (2008). Cookbook formulae for audio eq biquad filter coefficients. Online resource, accessed 14.4.2008
[6] Cristianini, N.; Shawe-Taylor, J., An introduction to support vector machines and other kernel-based learning methods, (2000), Cambridge University Press · Zbl 0994.68074
[7] Feldkamp, L.; Prokhorov, D.; Eagen, C.; Yuan, F., Enhanced multi-stream Kalman filter training for recurrent networks, Nonlinear modeling: advanced black-box techniques, 29-53, (1998)
[8] Gers, F.A.; Eck, D.; Schmidhuber, J., Applying LSTM to time series predictable through time-window approaches, Lecture notes in computer science, 2130, 669+, (2001) · Zbl 1005.68945
[9] Godsill, S.; Rayner, P., Digital audio restoration, (1997), Springer · Zbl 0930.94003
[10] Goodman, D.; Lockhart, G.; Wasern, O.; Wai-Choong, W., Waveform substitution techniques for recovering missing speech segments in packet voice communications, Acoustics, speech, and signal processing, IEEE transactions, 34, 1440-1448, (1986)
[11] Huang, Y.; Benesty, J.; Chen, J., Acoustic MIMO signal processing, (2006), Springer-Verlag Berlin, Heidelberg · Zbl 1106.94003
[12] Jaeger, H. (2001). The “echo state” approach to analysing and training recurrent neural networks. Tech. rep. no. 148. German National Research Center for Information Technology Bremen; Jaeger, H. (2001). The “echo state” approach to analysing and training recurrent neural networks. Tech. rep. no. 148. German National Research Center for Information Technology Bremen
[13] Jaeger, H. (2002a). Short-term memory in echo state networks. Tech. rep. no. 152. German National Research Center for Information Technology Bremen; Jaeger, H. (2002a). Short-term memory in echo state networks. Tech. rep. no. 152. German National Research Center for Information Technology Bremen
[14] Jaeger, H. (2002b). A tutorial on training recurrent neural networks, covering bppt, rtrl, ekf and the “echo state network” approach. Tech. rep. no. 159. German National Research Center for Information Technology Bremen; Jaeger, H. (2002b). A tutorial on training recurrent neural networks, covering bppt, rtrl, ekf and the “echo state network” approach. Tech. rep. no. 159. German National Research Center for Information Technology Bremen
[15] Jaeger, H., Adaptive nonlinear system identification with echo state networks, (), 593-600
[16] Jaeger, H.; Hass, H., Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication, Science, 304, 78-80, (2004)
[17] Jaeger, H.; Lukoševičius, M.; Popovici, D.; Siewert, U., 2007 special issue: optimization and applications of echo state networks with leaky-integrator neurons, Neural networks, 20, 3, 335-352, (2007) · Zbl 1132.68554
[18] Knapp, C.; Carter, G., The generalized correlation method for estimation of time delay, Acoustics, speech, and signal processing; IEEE transactions on signal processing, (1976)
[19] Maass, W.; Bishop, C., Pulsed neural networks, (2001), MIT Press
[20] Maass, W.; Natschlaeger, T.; Markram, H., Real-time computing without stable states: A new framework for neural computation based on perturbations, Neural computation, 14, 11, 2531-2560, (2002) · Zbl 1057.68618
[21] Niedzwiecki, M.; Cisowski, K., Smart copying — a new approach to reconstruction of audio signals, IEEE transactions on signal processing, 49, 2272-2282, (2001)
[22] Paugam-Moisy, H.; Martinez, R.; Bengio, S., Delay learning and polychronization for reservoir computing, Neurocomputing, 71, 7-9, 1143-1158, (2008)
[23] Pedersen, K. I., Fleury, B. H., & Mogensen, P. E. (1997). High resolution of electromagnetic waves in time-varying radiochannels. In Personal, indoor and mobile radio communications, 1997. ‘Waves of the year 2000’, Vol. 2 (pp. 650-654); Pedersen, K. I., Fleury, B. H., & Mogensen, P. E. (1997). High resolution of electromagnetic waves in time-varying radiochannels. In Personal, indoor and mobile radio communications, 1997. ‘Waves of the year 2000’, Vol. 2 (pp. 650-654)
[24] Schmidhuber, J.; Wierstra, D.; Gagliolo, M.; Gomez, F., Training recurrent networks by evolino, Neural computation, 19, 3, 757-779, (2007) · Zbl 1127.68085
[25] Siegelmann, H.T.; Sontag, E.D., Turing computability with neural nets, Applied mathematics letters, 4, 77-80, (1991) · Zbl 0749.68032
[26] Smith, J.O., Introduction to digital filters with audio applications, (2007), W3K Publishing, Online book, accessed 14.4.2008
[27] Smith, J.O., Mathematics of the discrete Fourier transform (dft), (2007), W3K Publishing, Online book, accessed 24.4.2008
[28] Swadlow, H., Physiological properties of individual cerebral axons studied in vivo for as long as one year, Journal of neurophysiology, 54, 1346-1362, (1985)
[29] Verstraeten, D.; Schrauwen, B.; D’Haene, M.; Stroobandt, D., 2007 special issue: an experimental unification of reservoir computing methods, Neural networks, 20, 3, 391-403, (2007) · Zbl 1132.68605
[30] Wustlich, W.; Siewert, U., Echo-state networks with band-pass neurons: towards generic time-scale-independent reservoir structures, PLANET intelligent systems gmbh, (2007), posted on the reservoir computing mailing list, accessed 23.3.2008
[31] Wyffels, F., Schrauwen, B., Verstraeten, D., & Stroobandt, D. (2008). Band-pass reservoir computing (pp. 3203-3208); Wyffels, F., Schrauwen, B., Verstraeten, D., & Stroobandt, D. (2008). Band-pass reservoir computing (pp. 3203-3208)
[32] Xue, Y.; Yang, L.; Haykin, S., 2007 special issue: decoupled echo state networks with lateral inhibition, Neural networks, 20, 3, 365-376, (2007) · Zbl 1132.68610
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.