×

zbMATH — the first resource for mathematics

Deep network based on stacked orthogonal convex incremental ELM autoencoders. (English) Zbl 1400.68185
Summary: Extreme learning machine (ELM) as an emerging technology has recently attracted many researchers’ interest due to its fast learning speed and state-of-the-art generalization ability in the implementation. Meanwhile, the incremental extreme learning machine (I-ELM) based on incremental learning algorithm was proposed which outperforms many popular learning algorithms. However, the incremental algorithms with ELM do not recalculate the output weights of all the existing nodes when a new node is added and cannot obtain the least-squares solution of output weight vectors. In this paper, we propose orthogonal convex incremental learning machine (OCI-ELM) with Gram-Schmidt orthogonalization method and Barron’s convex optimization learning method to solve the nonconvex optimization problem and least-squares solution problem, and then we give the rigorous proofs in theory. Moreover, in this paper, we propose a deep architecture based on stacked OCI-ELM autoencoders according to stacked generalization philosophy for solving large and complex data problems. The experimental results verified with both UCI datasets and large datasets demonstrate that the deep network based on stacked OCI-ELM autoencoders (DOC-IELM-AEs) outperforms the other methods mentioned in the paper with better performance on regression and classification problems.
MSC:
68T05 Learning and adaptive systems in artificial intelligence
62J05 Linear regression; mixed models
Software:
darch
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Huang, G.-B.; Zhu, Q.-Y.; Siew, C.-K., Extreme learning machine: theory and applications, Neurocomputing, 70, 1–3, 489-501, (2006)
[2] Huang, G.-B.; Wang, D. H.; Lan, Y., Extreme learning machines: a survey, International Journal of Machine Learning and Cybernetics, 2, 2, 107-122, (2011)
[3] Wang, X.-Z.; Shao, Q.-Y.; Miao, Q.; Zhai, J.-H., Architecture selection for networks trained with extreme learning machine using localized generalization error model, Neurocomputing, 102, 3-9, (2013)
[4] Fu, A. M.; Dong, C. R.; Wang, L. S., An experimental study on stability and generalization of extreme learning machines, International Journal of Machine Learning and Cybernetics, 6, 1, 129-135, (2015)
[5] Wang, X.-Z.; Ashfaq, R. A. R.; Fu, A.-M., Fuzziness based sample categorization for classifier performance improvement, Journal of Intelligent and Fuzzy Systems, 29, 3, 1185-1196, (2015)
[6] Wu, J.; Wang, S. T.; Chung, F.-L., Positive and negative fuzzy rule system, extreme learning machine and image classification, International Journal of Machine Learning and Cybernetics, 2, 4, 261-271, (2011)
[7] Lu, S.; Wang, X.; Zhang, G.; Zhou, X., Effective algorithms of the Moore-Penrose inverse matrices for extreme learning machine, Intelligent Data Analysis, 19, 4, 743-760, (2015)
[8] Huang, G.-B.; Chen, L.; Siew, C.-K., Universal approximation using incremental constructive feedforward networks with random hidden nodes, IEEE Transactions on Neural Networks, 17, 4, 879-892, (2006)
[9] Zhang, J.; Ding, S.; Zhang, N.; Shi, Z., Incremental extreme learning machine based on deep feature embedded, International Journal of Machine Learning and Cybernetics, 7, 1, 111-120, (2016)
[10] Ye, Y.; Qin, Y., QR factorization based Incremental Extreme Learning Machine with growth of hidden nodes, Pattern Recognition Letters, 65, 177-183, (2015)
[11] Ding, J.-L.; Wang, F.; Sun, H.; Shang, L., Improved incremental regularized extreme learning machine algorithm and its application in two-motor decoupling control, Neurocomputing, 149, 215-223, (2015)
[12] Xu, Z.; Yao, M.; Wu, Z.; Dai, W., Incremental regularized extreme learning machine and it’s enhancement, Neurocomputing, 174, 134-142, (2016)
[13] Huang, G.-B.; Li, M.-B.; Chen, L.; Siew, C.-K., Incremental extreme learning machine with fully complex hidden nodes, Neurocomputing, 71, 4–6, 576-583, (2008)
[14] Li, Y., Orthogonal incremental extreme learning machine for regression and multiclass classification, Neural Computing & Applications, 27, 1, 111-120, (2016)
[15] Huang, G.-B.; Chen, L., Convex incremental extreme learning machine, Neurocomputing, 70, 16–18, 3056-3062, (2007)
[16] Hinton, G. E.; Salakhutdinov, R. R., Reducing the dimensionality of data with neural networks, Science, 313, 5786, 504-507, (2006) · Zbl 1226.68083
[17] Ding, S.; Zhang, N.; Xu, X.; Guo, L.; Zhang, J., Deep extreme learning machine and its application in EEG classification, Mathematical Problems in Engineering, 2015, (2015)
[18] Vinyals, O.; Jia, Y.; Deng, L.; Darrell, T., Learning with recursive perceptual representations, Advances in Neural Information Processing Systems, 2825-2833, (2012)
[19] Krizhevsky, A.; Sutskever, I.; Hinton, G. E., ImageNet classification with deep convolutional neural networks, Advances in Neural Information Processing Systems, 2, (2012), MIT Press
[20] Socher, R.; Pennington, J.; Huang, E. H.; Ng, A. Y.; Manning, C. D., Semi-supervised recursive autoencoders for predicting sentiment distributions, Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP ’11), Association for Computational Linguistics
[21] Bengio, Y.; Delalleau, O.; Kivinen, J.; Szepesvári, C.; Ukkonen, E.; Zeugmann, T., On the expressive power of deep architectures, Algorithmic Learning Theory. Algorithmic Learning Theory, Lecture Notes in Computer Science, 6925, 18-36, (2011), New York, NY, USA: Springer, New York, NY, USA · Zbl 1348.68183
[22] Bengio, Y., Learning deep architectures for AI, Foundations and Trends in Machine Learning, 2, 1, 1-27, (2009) · Zbl 1192.68503
[23] Bengio, Y.; Lecun, Y., Scaling learning algorithms towards AI, Large-Scale Kernel Machines, 2007, 34, 1-41, (2007)
[24] Shores, T. S., Applied Linear Algebra and Matrix Analysis, (2007), Berlin, Germany: Springer, Berlin, Germany · Zbl 1128.15001
[25] Taguchi, G.; Jugulum, R., The Mahalanobis Taguchi Strategy: A Pattern Technology System, (2002), Hoboken, NJ, USA: John Wiley & Sons, Hoboken, NJ, USA
[26] Yang, Y. M.; Wang, Y. N.; Yuan, X. F., Parallel chaos search based incremental extreme learning machine, Neural Processing Letters, 37, 3, 277-301, (2013)
[27] Yu, Q.; Miche, Y.; Séverin, E.; Lendasse, A., Bankruptcy prediction using Extreme Learning Machine and financial expertise, Neurocomputing, 128, 296-302, (2014)
[28] Wong, K. I.; Chi, M. V.; Wong, P. K., Sparse Bayesian extreme learning machine and its application to biofuel engine performance prediction, Neurocomputing, 2015, 149, 397-404, (2015)
[29] Kasun, L. L. C.; Zhou, H.; Huang, G. B.; Vong, C. M., Representational learning with extreme learning machine, IEEE Intelligent Systems, 6, 28, 31-34, (2013)
[30] Johnson, W.; Lindenstrauss, J., Extensions of Lipschitz maps into a Hilbert space, Modern Analysis and Probability, 189, 26, 189-206, (1984) · Zbl 0539.46017
[31] Hinton, G. E.; Osindero, S.; Teh, Y.-W., A fast learning algorithm for deep belief nets, Neural Computation, 18, 7, 1527-1554, (2006) · Zbl 1106.68094
[32] Hinton, G. E., A practical guide to training restricted Boltzmann machines, Momentum, 1, 9, 599-619, (2010)
[33] Salakhutdinov, R.; Larochelle, H., Efficient learning of deep Boltzmann machines, Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS ’10)
[34] Zhou, H.; Huang, G. B.; Lin, Z., Stacked extreme learning machines, IEEE Transactions on Cybernetics, 2, 2, 1-13, (2014)
[35] Hearst, M. A.; Dumais, S. T.; Osman, E.; Platt, J.; Scholkopf, B., Support vector machines, IEEE Intelligent Systems, 13, 4, 18-28, (1998)
[36] Yu, H.; Reiner, P. D.; Xie, T.; Bartczak, T.; Wilamowski, B. M., An incremental design of radial basis function networks, IEEE Transactions on Neural Networks and Learning Systems, 25, 10, 1793-1803, (2014)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.