×

Parameter selection of Gaussian kernel SVM based on local density of training set. (English) Zbl 07402873

Summary: Support vector machine (SVM) is regarded as one of the most effective techniques for supervised learning, while the Gaussian kernel SVM is widely utilized due to its excellent performance capabilities. To ensure high performance of models, hyperparameters, i.e. kernel width and penalty factor must be determined appropriately. This paper studies the influence of hyperparameters on the Gaussian kernel SVM when such hyperparameters attain an extreme value (0 or \(\infty)\). In order to improve computing efficiency, a parameter optimization method based on the local density and accuracy of Leave-One-Out (LOO) method are proposed. Kernel width of each sample is determined based on the local density needed to ensure a higher separability in feature space while the penalty parameter is determined by an improved grid search using the LOO method. A comparison with grid method is conducted to verify validity of the proposed method. The classification accuracy of five real-life datasets from UCI database are 0.9733, 0.9933, 0.7270, 0.6101 and 0.8867, which is slightly superior to the grid method. The results also demonstrate that this proposed method is computationally cheaper by 1 order of magnitude when compared to the grid method.

MSC:

68Txx Artificial intelligence
62H30 Classification and discrimination; cluster analysis (statistical aspects)
PDFBibTeX XMLCite
Full Text: DOI Link

References:

[1] Vapnik VN. An overview of statistical learning theory. IEEE Trans Neural Netw. 1999; 10(5):988-999. · doi:10.1109/72.788640
[2] Mercer J. Functions of positive and negative type, and their connection with the theory of integral equations. Proc R Soc London, Ser A. 1909;83(559):69-70. · JFM 40.0408.02 · doi:10.1098/rspa.1909.0075
[3] Al-Mejibli IS, Abd DH, Alwan JK, et al. Performance evaluation of kernels in support vector machine; Fallujah, Iraq: 2018 1st Annual International Conference on Information and Sciences (AiCIS); 2018. · doi:10.1109/AiCIS.2018.00029
[4] Kaestner CAA. Support vector machines and kernel functions for text processing. Revista de Informática Teórica e Aplicada. 2013;20:130-154. · doi:10.22456/2175-2745.39702
[5] Dai X, Wang N, Wang W. Application of machine learning in BGP anomaly detection. J Phys Conf Ser. 2019;1176:032015. · doi:10.1088/1742-6596/1176/3/032015
[6] Fadel S, Ghoniemy S, Abdallah M, et al. Investigating the effect of different kernel functions on the performance of SVM for recognizing arabic characters. Int J Adv Comp Sci Appl. 2016;7 (1):446-450.
[7] Kumar D, Kumar R. Spam filtering using SVM with different kernel functions. Int J Comput Appl. 2016;136(5):16-23.
[8] Sutao S, Zhichao Z, Zhiying L, et al. Comparative study of SVM methods combined with voxel selection for object category classification on fMRI data. Plos One. 2011;6(2):e17191. · doi:10.1371/journal.pone.0017191
[9] Fan J, Jing F, Fang Z, et al. Automatic recognition system of welding seam type based on SVM method. Int J Adv Manuf Technol. 2017;92(1):989-999. · doi:10.1007/s00170-017-0202-8
[10] Hsu CW, Lin CJ. A simple decomposition method for support vector machines. Mach Learn. 2002;46(1-3):291-314. · Zbl 0998.68108 · doi:10.1023/A:1012427100071
[11] Lavalle SM, Branicky MS. On the relationship between classical grid search and probabilistic roadmaps. Int J Rob Res. 2003;23(23):673-692.
[12] Azimirad V, Hajibabzadeh M, Shahabi P. A new brain-robot interface system based on SVM-PSO classifier; Shiraz, Iran: Artificial Intelligence & Signal Processing Conference; 2017. · doi:10.1109/AISP.2017.8324122
[13] Gao Y, Leng J, Li S. Residual life prediction method for remanufacturing sucker rods based on magnetic memory testing and a support vector machine model. Insight. 2019;61(1):44-50. · doi:10.1784/insi.2019.61.1.44
[14] Manurung J, Mawengkang H, Zamzami E. Optimizing support vector machine parameters with genetic algorithm for credit risk assessment[J]. J Phys Conf Ser. 2017;930:012026. · doi:10.1088/1742-6596/930/1/012026
[15] Niu G-C, Wang Y, Hu Z, et al. Application of AHP and EIE in reliability analysis of complex production lines systems. Math Probl Eng. 2019;2019:1-10. · Zbl 1435.93102 · doi:10.1155/2019/3596784
[16] Subasi A. Classification of EMG signals using PSO optimized SVM for diagnosis of neuromuscular disorders. Comp Biol Med. 2013;43(5):576-586. · doi:10.1016/j.compbiomed.2013.01.020
[17] Xiaomu Z, Tao W, Jianjun H, et al. Study on tariff risk early warning of electric power users based on PSO-SVM algorithm; Beijing, China: 2018 International Conference on Big Data And Artificial Intelligence(BDAI); 2018. · doi:10.1109/BDAI.2018.8546635
[18] Huang CL, Wang CJ. A GA-based feature selection and parameters optimizationfor support vector machines. Expert Syst Appl. 2006;31(2):231-240. · doi:10.1016/j.eswa.2005.09.024
[19] Liu Y, Shi YK, Xu MW, et al. A further improved support vector machine model along with particle swarm optimization for face orientations recognition based on eigeneyes by using hybrid kernel; Bangkok, Thailand: IEEE International Conference on Industrial Engineering & Engineering Management; 2018.
[20] Niu D, Zhao W, Si L, et al. Cost forecasting of substation projects based on cuckoo search algorithm and support vector machines. Sustainability. 2018;10(1):118-128. · doi:10.3390/su10010118
[21] Nejad MB, Ahmadabadi MES. A novel image categorization strategy based on salp swarm algorithm to enhance efficiency of MRI images. Comput Model Eng Sci. 2019;119(1):185-205.
[22] Kilany M, Houssein EH, Hassanien AE, et al. Hybrid water wave optimization and support vector machine to improve EMG signal classification for neurogenic disorders; Cairo, Egypt: 2017 12th International Conference on Computer Engineering and Systems (ICCES); 2017. · doi:10.1109/ICCES.2017.8275392
[23] Yang D, Liu Y, Li S, et al. Gear fault diagnosis based on support vector machine optimized by artificial bee colony algorithm. Mech Mach Theor. 2015;90:219-229. · doi:10.1016/j.mechmachtheory.2015.03.013
[24] Anis S, El-Mahlawy MH, Gadallah MEA, et al. Parametric fault detection of analogue circuits. Int J Comput Appl. 2014;96(9):14-23.
[25] Injadat M, Salo F, Nassif AB, et al. Bayesian optimization with machine learning algorithms towards anomaly detection; Abu Dhabi, UAE: 2018 IEEE global Communications Conference (GLOBECOM); 2018. · doi:10.1109/GLOCOM.2018.8647714
[26] Keerthi SS, Lin C-J. Asymptotic behaviors of support vector machines with Gaussian kernel. Neural Comput. 2003;15(7):1667-1689. · Zbl 1086.68569 · doi:10.1162/089976603321891855
[27] Shi H, Xiao H, Zhou J, et al. Radial basis function kernel parameter optimization algorithm in support vector machine based on segmented dichotomy; Nanjing, China: 2018 5th International Conference on Systemsand Informatics (ICSAI); 2018. · doi:10.1109/ICSAI.2018.8599461
[28] Han Y, Li J, Li JZ, et al. Demonstration of SVM Classification Based on Improved Gauss Kernel Function; Beijing, China: First International Conference on Real Time Intelligent Systems; 2016.
[29] Wang D, Xiang-Bin WU, Lin DM. Two heuristic strategies for searching optimal hyper parameters of C-SVM; Baoding, China: International Conferenceon on Machine Learning & Cybernetics; 2009. · doi:10.1109/ICMLC.2009.5212727
[30] Xie Y, Bian C, Lu Murphey Y, et al. An SVM parameter learning algorithm scalable on large data size for driver fatigue detection; Honolulu, Hawaii, USA: 2017 IEEE Symposium Series on Computational Intelligence; 2017. · doi:10.1109/SSCI.2017.8285380
[31] Xie YQ, Murphey YL, Kochhar DS. SVM parameter optimization using swarm intelligence for learning from big data. In: Nguyen NT, Pimenidis E, Khan Z, editors. Computational Collective intelligence, ICCCI 2018, Pt I. New York: Springer; 2018. p. 469-478. · doi:10.1007/978-3-319-98443-8_43
[32] Li C, Hsieh P, Kuo B. Multiple SVMS based on random subspaces from kernel feature importance for hyperspectral image classification; Fort Worth, Texas, USA: 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS); 2017. · doi:10.1109/IGARSS.2017.8127018
[33] Kuo B, Ho H, Li C, et al. A kernel-based feature selection method for SVM With RBF kernel for hyperspectral image classification. IEEE J Sel Top Appl Earth Obs Remote Sens. 2014;7(1):317-326. · doi:10.1109/JSTARS.2013.2262926
[34] Li CH, Lin CT, Kuo BC, et al. An automatic method for selecting the parameter of the normalized kernel function to support vector machines; Hsinchu City, Taiwan: International Conference on Technologies & Applications of Artificial Intelligence; 2010. · doi:10.1109/TAAI.2010.46
[35] Qu L, Zhou H, Liu C, et al. Study on Multi-RBF-SVM for Transformer Fault Diagnosis; Wuxi, China: 2018 17th International Symposium on Distributed Computing and Applications for Business Engineering and Science (DCABES); 2018. · doi:10.1109/DCABES.2018.00056
[36] Gehler PV, Schölkopf B. An introduction to kernel learning algorithms. Hoboken (NJ): John Wiley & Sons Inc.; 2009. · doi:10.1002/9780470748992.ch2
[37] Huang Y, Zheng CY, Song ZH. Multi-class support vector machines algorithm summarization. Comp Technol Autom. 2005;24(4):61-63.
[38] Melgani F, Bruzzone L. Classification of hyperspectral remote sensing images with support vector machines. IEEE Trans Geosci Remote Sens. 2004;42(8):1778-1790. · doi:10.1109/TGRS.2004.831865
[39] Bottou L., Cortes C., Denker J.S., et al. Comparison of classifier methods: a case study in handwritten digit recognition; Jerusalem, Israel: International Conference on Pattern Recognition; 1994. · doi:10.1109/ICPR.1994.576879
[40] Lin CJ. Formulations of support vector machines: a note from an optimization point of view. Neural Comput. 2001;13(2):307-317. · Zbl 0973.68201 · doi:10.1162/089976601300014547
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.