×

Functional echo state network for time series classification. (English) Zbl 1429.68256

Summary: Echo state networks (ESNs) are a new approach to recurrent neural networks (RNNs) that have been successfully applied in many domains. Nevertheless, an ESN is a predictive model rather than a classifier, and methods to employ ESNs in time series classification (TSC) tasks have not yet been fully explored. In this paper, we propose a novel ESN approach named functional echo state network (FESN) for time series classification. The basic idea behind FESN is to replace the numeric variable output weights of an ESN with time-varying output-weight functions and introduce a temporal aggregation operator to the output layer that can project temporal signals into discrete class labels, thereby transforming the ESN from a predictive model into a true classifier. Subsequently, to learn the output-weight functions, a spatio-temporal aggregation learning algorithm is proposed based on orthogonal function basis expansion. By leveraging the nonlinear mapping capacity of a reservoir and the accumulation of temporal information in the time domain, FESN can not only enhance the separability of different classes in a high-dimensional functional space but can also consider the relative importance of temporal data at different time steps according to dynamic output-weight functions. Theoretical analyses and experiments on an extensive set of UCR data were conducted on FESN. The results show that FESN yields better performance than single-algorithm methods, has comparable accuracy with ensemble-based methods and exhibits acceptable computational complexity. Interestingly, for some time series datasets, we visualized some interpretable features extracted by FESN via specific patterns within the output-weight functions.

MSC:

68T07 Artificial neural networks and deep learning
62H30 Classification and discrimination; cluster analysis (statistical aspects)
62M10 Time series, auto-correlation, regression, etc. in statistics (GARCH)
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Bagnall, A.; Bostrom, A.; Large, J.; Lines, J., The great time series classification bake off: An experimental evaluation of recently proposed algorithms. extended version (2016), arXiv:1602.01711
[2] Bagnall, A.; Davis, L. M.; Hills, J.; Lines, J., Transformation based ensembles for time series classification., SDM, 12, 307-318 (2012), SIAM
[3] Bagnall, A.; Lines, J.; Hills, J.; Bostrom, A., Time-series classification with cote: the collective of transformation-based ensembles, IEEE Trans. Knowl. Data Eng., 27, 9, 2522-2535 (2015)
[4] Batista, G. E.; Wang, X.; Keogh, E. J., A complexity-invariant distance measure for time series., SDM, 11, 699-710 (2011), SIAM
[5] Baydogan, M. G.; Runger, G.; Tuv, E., A bag-of-features framework to classify time series, IEEE Trans. Pattern Anal. Mach. Intell., 35, 11, 2796-2802 (2013)
[6] Berndt, D. J.; Clifford, J., Using dynamic time warping to find patterns in time series., KDD workshop, 10, 359-370 (1994), Seattle, WA
[7] Büsing, L.; Schrauwen, B.; Legenstein, R., Connectivity, dynamics, and memory in reservoir computing with binary and analog neurons, Neural comput., 22, 5, 1272-1311 (2010) · Zbl 1195.92002
[8] Chan, K.-P.; Fu, A. W.-C., Efficient time series matching by wavelets, Data Engineering, 1999. Proceedings., 15th International Conference on, 126-133 (1999), IEEE
[9] Chaovalitwongse, W. A.; Pottenger, R. S.; Wang, S.; Fan, Y.-J.; Iasemidis, L. D., Pattern-and network-based classification techniques for multichannel medical data signals to improve brain diagnosis, IEEE Trans. Syst. Man Cybern. Part A, 41, 5, 977-988 (2011)
[10] Chatzis, S. P.; Demiris, Y., Echo state gaussian process, IEEE Trans. Neural Netw., 22, 9, 1435-1445 (2011)
[11] Chen, H.; Tang, F.; Tino, P.; Cohn, A. G.; Yao, X., Model metric co-learning for time series classification, Proceedings of the 24th International Conference on Artificial Intelligence, 3387-3394 (2015), AAAI Press
[12] Chen, L.; Ng, R., On the marriage of lp-norms and edit distance, Proceedings of the Thirtieth international conference on Very large data bases-Volume 30, 792-803 (2004), VLDB Endowment
[13] Chen, L.; Özsu, M. T.; Oria, V., Robust and fast similarity search for moving object trajectories, Proceedings of the 2005 ACM SIGMOD international conference on Management of data, 491-502 (2005), ACM
[15] Chen, Y.; Nascimento, M.; Ooi, B. C.; Tung, A. K., Spade: On shape-based pattern detection in streaming time series, Data Engineering, 2007. ICDE 2007. IEEE 23rd International Conference on, 786-795 (2007), IEEE
[16] Chen, Z.; Wangmeng, Z.; Hu, Q.; Liang, L., Kernel sparse representation for time series classification, Inf. Sci., 292, 15-26 (2015) · Zbl 1404.68095
[17] Costa, I. G.; Schönhuth, A.; Hafemeister, C.; Schliep, A., Constrained mixture estimation for analysis and robust classification of clinical time series, Bioinformatics, 25, 12, i6-i14 (2009)
[18] Cover, T.; Hart, P., Nearest neighbor pattern classification, IEEE trans. inf. theory, 13, 1, 21-27 (1967) · Zbl 0154.44505
[19] Deihimi, A.; Showkati, H., Application of echo state networks in short-term electric load forecasting, Energy, 39, 1, 327-340 (2012)
[20] Demšar, J., Statistical comparisons of classifiers over multiple data sets, J. Mach. Learn. Res., 7, 1-30 (2006) · Zbl 1222.68184
[21] Deng, H.; Runger, G.; Tuv, E.; Vladimir, M., A time series forest for classification and feature extraction, Inf. Sci., 239, 142-153 (2013) · Zbl 1321.62068
[22] Ding, H.; Trajcevski, G.; Scheuermann, P.; Wang, X.; Keogh, E., Querying and mining of time series data: experimental comparison of representations and distance measures, Proc. VLDB Endowment, 1, 2, 1542-1552 (2008)
[23] Douzal-Chouakria, A.; Amblard, C., Classification trees for time series, Pattern Recog., 45, 3, 1076-1091 (2012)
[24] Faloutsos, C.; Ranganathan, M.; Manolopoulos, Y., Fast subsequence matching in time-series databases, 23 (1994), ACM
[25] Fink, O.; Zio, E.; Weidmann, U., Quantifying the reliability of fault classifiers, Inf. Sci., 266, 65-74 (2014)
[26] Frank, J.; Mannor, S.; Pineau, J.; Precup, D., Time series analysis using geometric template matching, IEEE Trans. Pattern Anal. Mach. Intell., 35, 3, 740-754 (2013)
[27] Frentzos, E.; Gratsias, K.; Theodoridis, Y., Index-based most similar trajectory search, Data Engineering, 2007. ICDE 2007. IEEE 23rd International Conference on, 816-825 (2007), IEEE
[28] Fu, T.; Law, C.; Chan, K.; Chung, F.; Ng, C., Stock time series categorization and clustering via sb-tree optimization, Fuzzy Systems and Knowledge Discovery, 1130-1139 (2006), Springer
[29] Fulcher, B. D.; Jones, N. S., Highly comparative feature-based time-series classification, IEEE Trans. Knowl. Data Eng., 26, 12, 3026-3037 (2014)
[30] García-Treviño, E. S.; Barria, J. A., Structural generative descriptions for time series classification, IEEE trans. cybern., 44, 10, 1978-1991 (2014)
[31] Geurts, P., Pattern extraction for time series classification, Principles of Data Mining and Knowledge Discovery, 115-127 (2001), Springer · Zbl 1009.68745
[32] Gordon, D.; Hendler, D.; Rokach, L., Fast and space-efficient shapelets-based time-series classification, Intell. Data Anal., 19, 5, 953-981 (2015)
[33] Górecki, T., Using derivatives in a longest common subsequence dissimilarity measure for time series classification, Pattern Recog. Lett., 45, 99-105 (2014)
[34] Górecki, T.; Łuczak, M., Non-isometric transforms in time series classification using dtw, Knowledge-Based Syst., 61, 98-108 (2014)
[35] He, X.; Xu, S., Process neural networks: theory and applications (2010), Springer Science & Business Media
[36] Jaeger, H., Controlling recurrent neural networks by conceptors (2014), arXiv:1403.3369
[37] Jaeger, H.; Haas, H., Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication, Science, 304, 5667, 78-80 (2004)
[38] Jeong, Y.-S.; Jeong, M. K.; Omitaomu, O. A., Weighted dynamic time warping for time series classification, Pattern Recog., 44, 9, 2231-2240 (2011)
[39] Kate, R. J., Using dynamic time warping distances as features for improved time series classification, Data Mining Knowl. Discovery, 1-30 (2015)
[40] Kaya, H.; Gündüz-Öğüdücü, Ş., A distance based time series classification framework, Inf. Syst., 51, 27-42 (2015)
[41] Legenstein, R.; Maass, W., Edge of chaos and prediction of computational performance for neural circuit models, Neural Netw., 20, 3, 323-334 (2007) · Zbl 1132.68568
[42] Li, D.; Han, M.; Wang, J., Chaotic time series prediction based on a novel robust echo state network, IEEE Trans. Neural Netw. Learn. Syst., 23, 5, 787-799 (2012)
[43] Lines, J.; Bagnall, A., Time series classification with ensembles of elastic distance measures, Data Mining Knowl. Discovery, 29, 3, 565-592 (2015) · Zbl 1405.68295
[44] Lines, J.; Davis, L. M.; Hills, J.; Bagnall, A., A shapelet transform for time series classification, Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, 289-297 (2012), ACM
[45] Liu, Y.; Liu, Q.; Wang, W.; Zhao, J.; Leung, H., Data-driven based model for flow prediction of steam system in steel industry, Inf. sci., 193, 104-114 (2012)
[46] LukošEvičIus, M.; Jaeger, H., Reservoir computing approaches to recurrent neural network training, Comput. Sci. Rev., 3, 3, 127-149 (2009) · Zbl 1302.68235
[47] Lun, S.; Yao, X.; Hu, H., A new echo state network with variable memory length, Inf. Sci., 370, 103-119 (2016) · Zbl 1428.68245
[48] Ma, Q.-L.; Chen, W.-B., Modular state space of echo state network, Neurocomputing, 122, 406-417 (2013)
[49] Morse, M. D.; Patel, J. M., An efficient and accurate method for evaluating time series similarity, Proceedings of the 2007 ACM SIGMOD international conference on Management of data, 569-580 (2007), ACM
[50] Noponen, K.; Kortelainen, J.; Seppänen, T., Invariant trajectory classification of dynamical systems with a case study on ecg, Pattern Recog., 42, 9, 1832-1844 (2009) · Zbl 1192.68533
[51] Orsenigo, C.; Vercellis, C., Combining discrete svm and fixed cardinality warping distances for multivariate time series classification, Pattern Recog., 43, 11, 3787-3794 (2010) · Zbl 1209.68422
[52] Rodan, A.; Tiňo, P., Simple deterministically constructed cycle reservoirs with regular jumps, Neural comput., 24, 7, 1822-1852 (2012)
[53] Skowronski, M. D.; Harris, J. G., Minimum mean squared error time series classification using an echo state network prediction model, Circuits and Systems, 2006. ISCAS 2006. Proceedings. 2006 IEEE International Symposium on, 3153-3156 (2006), IEEE
[54] Skowronski, M. D.; Harris, J. G., Automatic speech recognition using a predictive echo state network classifier, Neural netw., 20, 3, 414-423 (2007) · Zbl 1132.68663
[55] Verstraeten, D.; Schrauwen, B.; dHaene, M.; Stroobandt, D., An experimental unification of reservoir computing methods, Neural Netw., 20, 3, 391-403 (2007) · Zbl 1132.68605
[56] Vlachos, M.; Kollios, G.; Gunopulos, D., Discovering similar multidimensional trajectories, Data Engineering, 2002. Proceedings. 18th International Conference on, 673-684 (2002), IEEE
[57] Wang, L.; Wang, Z.; Liu, S., An effective multivariate time series classification approach using echo state network and adaptive differential evolution algorithm, Expert Syst. Appl., 43, 237-249 (2016)
[58] Wang, X.; Mueen, A.; Ding, H.; Trajcevski, G.; Scheuermann, P.; Keogh, E., Experimental comparison of representation methods and distance measures for time series data, Data Mining Knowl. Discovery, 26, 2, 275-309 (2013)
[59] Ye, L.; Keogh, E., Time series shapelets: a new primitive for data mining, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, 947-956 (2009), ACM
[60] Yu, D.; Yu, X.; Hu, Q.; Liu, J.; Wu, A., Dynamic time warping constraint learning for large margin nearest neighbor classification, Inf. Sci., 181, 13, 2787-2796 (2011)
[61] Yusoff, M.-H.; Chrol-Cannon, J.; Jin, Y., Modeling neural plasticity in echo state networks for classification and regression, Inf. Sci., 364, 184-196 (2016)
[62] Zheng, Y.; Liu, Q.; Chen, E.; Zhao, J. L.; He, L.; Lv, G., Convolutional nonlinear neighbourhood components analysis for time series classification, Advances in Knowledge Discovery and Data Mining, 534-546 (2015), Springer
[63] Zhu, L.; Lu, C.; Sun, Y., Time series shapelet classification based online short-term voltage stability assessment, IEEE Trans. Power Syst., 31, 2, 1430-1439 (2016)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.