×

Adaptively weighted large-margin angle-based classifiers. (English) Zbl 1499.62210

Summary: Large-margin classifiers are powerful techniques for classification problems. Although binary large-margin classifiers are heavily studied, multicategory problems are more complicated and challenging. A common approach is to construct \(k\) different decision functions for a \(k\)-class problem with a sum-to-zero constraint. However, such a constraint can be inefficient. Moreover, many large-margin classifiers can be sensitive to outliers in the training sample. In this article, we use the angle-based classification framework to avoid the explicit sum-to-zero constraint, and we propose two adaptively weighted large-margin classification techniques. Our new methods are Fisher consistent and more robust against outliers under suitable conditions. Numerical experiments further indicate that our methods give competitive and stable performance when compared with existing approaches.

MSC:

62H30 Classification and discrimination; cluster analysis (statistical aspects)
68T05 Learning and adaptive systems in artificial intelligence
PDFBibTeX XMLCite
Full Text: DOI Link

References:

[1] An, L. T.H.; Tao, P. D., Solving a class of linearly constrained indefinite quadratic problems by DC algorithms, J. Global Optim., 11, 253-285 (1997) · Zbl 0905.90131
[2] Bishop, C. M., Pattern Recognition and Machine Learning (2006), Springer: Springer New York · Zbl 1107.68072
[3] Boser, B. E.; Guyon, I. M.; Vapnik, V. N., A training algorithm for optimal margin classifiers, (Proceedings of the Fifth Annual Workshop on Computational Learning Theory (1992), ACM)
[4] Bredensteiner, E. J.; Bennett, K. P., Multicategory classification by support vector machines, (Computational Optimization (1999), Springer: Springer New York), 53-79 · Zbl 1040.90574
[5] Burges, C. J., A tutorial on support vector machines for pattern recognition, Data Min. Knowl. Discov., 2, 121-167 (1998)
[6] Cortes, C.; Vapnik, V., Support-vector networks, Mach. Learn., 20, 273-297 (1995) · Zbl 0831.68098
[7] Crammer, K.; Singer, Y., On the algorithmic implementation of multiclass kernel-based vector machines, J. Mach. Learn. Res., 2, 265-292 (2001) · Zbl 1037.68110
[8] Cristianini, N.; Shawe-Taylor, J., An Introduction To Support Vector Machines and Other Kernel-Based Learning Methods (2000), Cambridge University Press
[9] Dietterich, T. G.; Bakiri, G., Solving multiclass learning problems via error-correcting output codes, J. Artificial Intelligence Res., 263-286 (1995) · Zbl 0900.68358
[10] Fan, R.-E.; Chang, K.-W.; Hsieh, C.-J.; Wang, X.-R.; Lin, C.-J., Liblinear: A library for large linear classification, J. Mach. Learn. Res., 9, 1871-1874 (2008) · Zbl 1225.68175
[11] Hastie, T. J.; Tibshirani, R. J.; Friedman, J. H., The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2009), Springer: Springer New York · Zbl 1273.62005
[12] Kimeldorf, G. S.; Wahba, G., Some results on Tchebycheffian spline functions, J. Math. Anal. Appl., 33, 82-95 (1971) · Zbl 0201.39702
[13] Lee, Y.; Lin, Y.; Wahba, G., Multicategory support vector machines: Theory and application to the classification of microarray data and satellite radiance data, J. Amer. Statist. Assoc., 99, 67-81 (2004) · Zbl 1089.62511
[14] Y. Liu, Fisher consistency of multicategory support vector machines, in: International Conference on Machine Learning, 2007.; Y. Liu, Fisher consistency of multicategory support vector machines, in: International Conference on Machine Learning, 2007.
[15] Liu, Y.; Shen, X., Multicategory \(\psi \)-learning, J. Amer. Statist. Assoc., 101, 500-509 (2006) · Zbl 1119.62341
[16] Liu, Y.; Shen, X.; Doss, H., Multicategory \(\psi \)-learning and support vector machine: computational tools, J. Comput. Graph. Statist., 14, 219-236 (2005)
[17] Liu, Y.; Yuan, M., Reinforced multicategory support vector machines, J. Comput. Graph. Statist., 20, 901-919 (2011)
[18] Liu, Y.; Zhang, H. H.; Wu, Y., Hard or soft classification? Large-margin unified machines, J. Amer. Statist. Assoc., 106, 166-177 (2011) · Zbl 1396.62144
[19] Nesterov, Y., Efficiency of coordinate descent methods on huge-scale optimization problems, SIAM J. Optim., 22, 341-362 (2012) · Zbl 1257.90073
[20] Shen, X.; Tseng, G. C.; Zhang, X.; Wong, W. H., On \(\psi \)-learning, J. Amer. Statist. Assoc., 98, 724-734 (2003) · Zbl 1052.62095
[21] Tang, Y.; Zhang, H. H., Multiclass proximal support vector machines, J. Comput. Graph. Statist., 15, 339-355 (2006)
[22] Vapnik, V. N., Statistical Learning Theory, Vol. 1 (1998), Wiley: Wiley New York
[23] J. Weston, C. Watkins, Support vector machines for multi-class pattern recognition, in: ESANN’1999 Proceedings —European Symposium on Artificial Neural Networks Bruges (Belgium), 21-23 April 1999, pp. 219-224, D-Facto public, ISBN:2-600049-9-X.; J. Weston, C. Watkins, Support vector machines for multi-class pattern recognition, in: ESANN’1999 Proceedings —European Symposium on Artificial Neural Networks Bruges (Belgium), 21-23 April 1999, pp. 219-224, D-Facto public, ISBN:2-600049-9-X.
[24] Wu, Y.; Liu, Y., On multicategory truncated hinge loss support vector machines, (Verducci, J. S.; Shen, X.; Lafferty, J., Prediction and Discovery: AMS-IMS-SIAM Joint Summer Research Conference, Machine and Statistical Learning, Vol. 443 (2006), American Mathematical Society: American Mathematical Society Snowbird, Utah), 49-58 · Zbl 1145.62049
[25] Wu, Y.; Liu, Y., Robust truncated hinge loss support vector machines, J. Amer. Statist. Assoc., 102, 974-983 (2007) · Zbl 1469.62293
[26] Wu, Y.; Liu, Y., Adaptively weighted large margin classifiers, J. Comput. Graph. Statist., 22, 416-432 (2013)
[27] Zhang, T., Statistical analysis of some multi-category large margin classification methods, J. Mach. Learn. Res., 5, 1225-1251 (2004) · Zbl 1222.68344
[28] Zhang, C.; Liu, Y., Multicategory large-margin unified machines, J. Mach. Learn. Res., 14, 1349-1386 (2013) · Zbl 1317.68200
[29] Zhang, C.; Liu, Y., Multicategory angle-based large-margin classification, Biometrika, 101, 625-640 (2014) · Zbl 1335.62110
[30] Zhang, C.; Liu, Y.; Wang, J.; Zhu, H., Reinforced angle-based multicategory support vector machines, J. Comput. Graph. Statist., 25, 806-825 (2016)
[31] Zhang, C.; Pham, M.; Fu, S.; Liu, Y., Robust multicategory support vector machines using difference convex algorithm, Math. Program. (2018), (in press) · Zbl 1397.90319
[32] Zou, H.; Zhu, J.; Hastie, T., New multicategory boosting algorithms based on multicategory Fisher-consistent losses, Ann. Appl. Stat., 2, 1290-1306 (2008) · Zbl 1158.62044
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.