×

Direct kernel perceptron (DKP): ultra-fast kernel ELM-based classification with non-iterative closed-form weight calculation. (English) Zbl 1298.68219

Summary: The Direct Kernel Perceptron (DKP) is a very simple and fast kernel-based classifier, related to the Support Vector Machine (SVM) and to the Extreme Learning Machine (ELM), whose \(\alpha\)-coefficients are calculated directly, without any iterative training, using an analytical closed-form expression which involves only the training patterns. The DKP, which is inspired by the Direct Parallel Perceptron uses a Gaussian kernel and a linear classifier (perceptron). The weight vector of this classifier in the feature space minimizes an error measure which combines the training error and the hyperplane margin, without any tunable regularization parameter. This weight vector can be translated, using a variable change, to the \(\alpha\)-coefficients, and both are determined without iterative calculations. We calculate solutions using several error functions, achieving the best trade-off between accuracy and efficiency with the linear function. These solutions for the \(\alpha\) coefficients can be considered alternatives to the ELM with a new physical meaning in terms of error and margin: in fact, the linear and quadratic DKP are special cases of the two-class ELM when the regularization parameter \(C\) takes the values \(C=0\) and \(C=\infty\). The linear DKP is extremely efficient and much faster (over a vast collection of 42 benchmark and real-life data sets) than 12 very popular and accurate classifiers including SVM, Multi-Layer Perceptron, Adaboost, Random Forest and Bagging of RPART decision trees, Linear Discriminant Analysis, K-Nearest Neighbors, ELM, Probabilistic Neural Networks, Radial Basis Function neural networks and Generalized ART. Besides, despite its simplicity and extreme efficiency, DKP achieves higher accuracies than 7 out of 12 classifiers, exhibiting small differences with respect to the best ones (SVM, ELM, Adaboost and Random Forest), which are much slower. Thus, the DKP provides an easy and fast way to achieve classification accuracies which are not too far from the best one for a given problem. The C and Matlab code of DKP are freely available.

MSC:

68T05 Learning and adaptive systems in artificial intelligence
PDFBibTeX XMLCite
Full Text: DOI

References:

[2] Auer, P.; Burgsteiner, H.; Maass, W., A learning rule for very simple universal approximators consisting of a single layer of perceptrons, Neural Networks, 1, 21, 786-795 (2008) · Zbl 1254.68196
[3] Breiman, L., Bagging predictors, Machine Learning, 24, 2, 123-140 (1996) · Zbl 0858.68080
[4] Breiman, L., Random forests, Machine Learning, 45, 1, 5-32 (2001) · Zbl 1007.68152
[5] Carrión, P.; Cernadas, E.; Gálvez, J. F.; Sá-Otero, P., Classification of honeybee pollen using a multiscale texture filtering scheme, Machine Vision and Applications, 15, 186-193 (2004)
[7] Cortes, C.; Vapnik, V., Support-vector networks, Machine Learning, 20, 3, 273-297 (1995) · Zbl 0831.68098
[8] Dacal-Nieto, A.; Formella, A.; Carrión, P.; Vázquez-Fernández, E.; Fernández-Delgado, M., Common scab detection on potatoes using an infrared hyperspectral imaging, (Proc. int. conf. on image analysis and processing. Proc. int. conf. on image analysis and processing, LNCS, Vol. 6979 (2011), Springer), 303-312
[9] Delogu, R.; Fanni, A.; Montisci, A., Geometrical synthesis of MLP neural networks, Neurocomputing, 71, 919-930 (2008)
[11] Fernández-Delgado, M.; Ribeiro, J.; Cernadas, E.; Barro, S., Direct parallel perceptrons (DPPs): fast analytical calculation of the parallel perceptrons weights with margin control for classification tasks, IEEE Transactions on Neural Networks, 22, 1837-1848 (2011)
[13] González-Rufino, E.; Carrión, P.; Cernadas, E.; Fernández-Delgado, M.; Domínguez-Petit, R., Exhaustive comparison of colour texture features and classification methods to discriminate cells categories in histological images of fish ovary, Pattern Recognition, 46, 2391-2407 (2013)
[14] Haykin, S., Neural networks: a comprehensive foundation (1994), Macmillan Coll Div · Zbl 0828.68103
[15] Hsu, C.-W.; Lin, C.-J., A comparison of methods for multiclass support vector machines, IEEE Transactions on Neural Networks, 13, 2, 415-425 (2002)
[17] Huang, G.-B.; Chen, L., Enhanced random search based incremental extreme learning machine, Neurocomputing, 71, 3460-3468 (2008)
[18] Huang, G.-B.; Ding, X.; Zhou, H., Optimization method based extreme learning machine for classification, Neurocomputing, 74, 155-163 (2010)
[19] Huang, G.-B.; Wang, D.; Lan, Y., Extreme learning machines: a survey, International Journal of Machine Learning and Cybernetics, 2, 107-122 (2011)
[20] Huang, G.-B.; Zhou, H.; Ding, X.; Zhang, R., Extreme learning machine for regression and multiclass classification, IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics), 42, 513-529 (2012)
[21] Iglesias, R.; Fernández-Delgado, M.; Barciela, J.; Peña, R.; García-Martín, S.; Herrero, C., Comparison of several chemometric techniques for the classification of Orujo distillate alcoholic samples from Galicia (NW Spain) according to their certified brand of origin, Analytical and Bioanalytical Chemistry, 397, 6, 2603-2614 (2010)
[24] Martinetz, T.; Labusch, K.; Schneegaß, D., SoftDoubleMaxMinOver: perceptron-like training of Support Vector Machines, IEEE Transactions on Neural Networks, 20, 7, 1061-1072 (2009)
[25] Miche, Y.; van Heeswijk, M.; Bas, P.; Simula, O.; Lendasse, A., TROP-ELM: a double-regularized ELM using LARS and Tikhonov regularization, Neurocomputing, 74, 16, 2413-2421 (2011)
[27] Panagiotakopoulos, C.; Tsampouka, P., The margitron: a generalized perceptron with margin, IEEE Transactions on Neural Networks, 22, 3, 395-407 (2011)
[28] Peng, X.; Wang, Y., Geometric algorithms to large margin classifier based on affine hulls, IEEE Transactions on Neural Networks, 23, 2, 236-246 (2012)
[29] Rodríguez-Damián, M.; Cernadas, E.; Formella, A.; Fernández-Delgado, M.; Sá-Otero, P., Automatic detection and classification of grains of pollen based on shape and texture, IEEE Transactions on Systems, Man and Cybernetics, Part C, 36, 531-542 (2006)
[30] Sheskin, D., Handbook of parametric and nonparametric statistical procedures (2006), CRC Press
[31] Specht, D., Probabilistic neural networks, Neural Networks, 3, 1, 109-118 (1990)
[32] Talukder, A.; Casasent, D., A closed-form neural network for discriminatory feature extraction from high-dimensional data, Neural Networks, 14, 1201-1218 (2001)
[34] Toh, K.-A., An error-counting network for pattern classification, Neurocomputing, 71, 1680-1693 (2008)
[35] Venables, W.; Ripley, B., Modern applied statistics with S (2002), Springer · Zbl 1006.62003
[36] Wang, X.; Shao, Q.; Qing, M.; Zhai, J., Architecture selection for networks trained with extreme learning machine using localized generalization error model, Neurocomputing, 102, 3-9 (2013)
[37] Yap, K. S.; Lim, C. P.; Abidin, I. Z., A hybrid ART-GRNN online learning neural network with an \(\varepsilon \)-insensitive loss function, IEEE Transactions on Neural Networks, 19, 9, 1641-1646 (2008)
[38] Yujian, L.; Bo, L.; Xinwu, Y.; Yaozong, F.; Houjun, L., Multiconlitron: a general piecewise linear classifier, IEEE Transactions on Neural Networks, 22, 2, 276-289 (2011)
[39] Zhai, J.; Xu, H.; Wang, X., Dynamic ensemble extreme learning machine based on sample entropy, Soft Computing, 16, 9, 1493-1502 (2012)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.