zbMATH — the first resource for mathematics

Multi-class pattern classification using neural networks. (English) Zbl 1103.68777
Summary: Multi-class pattern classification has many applications including text document classification, speech recognition, object recognition, etc. Multi-class pattern classification using neural networks is not a trivial extension from two-class neural networks. This paper presents a comprehensive and competitive study in multi-class neural learning with focuses on issues including neural network architecture, encoding schemes, training methodology and training time complexity. Our study includes multi-class pattern classification using either a system of multiple neural networks or a single neural network, and modeling pattern classes using one-against-all, one-against-one, one-against-higher-order, and P-against-Q. We also discuss implementations of these approaches and analyze training time complexity associated with each approach. We evaluate six different neural network system architectures for multi-class pattern classification along the dimensions of imbalanced data, large number of pattern classes, large vs. small training data through experiments conducted on well-known benchmark data.

68T10 Pattern recognition, speech recognition
Full Text: DOI
[1] Le Cun, Y.; Boser, B.; Denker, J.; Hendersen, D.; Howard, R.; Hubbard, W.; Jackel, L., Backpropagation applied to handwritten zip code recognition, Neural comput., 1, 4, 541-551, (1989)
[2] Aha, D.; Bankert, R., Cloud classification using error-correcting output codes, Artif. intell. appl. nat. resour. agric. environ. sci., 11, 1, 13-28, (1997)
[3] Y.L. Murphey, Y. Luo, Feature extraction for a multiple pattern classification neural network system, IEEE International Conference on Pattern Recognition, August 2002.
[4] E. Brill, Some advances in transformation-based part of speech tagging, In: AAAI Conference, vol. 1, 1994, pp. 722-727.
[5] Y. Even-Zohar, D. Roth, A sequential model for multiclass classification, in: SIGDAT Conference on Empirical Methods in Natural Language Processing, 2001, pp. 10-19.
[6] Baldi, P.; Pollastri, B., A machine learning strategy for protein analysis, IEEE intell. syst., 17, 2, 28-35, (2002)
[7] Clare, A.; King, R.D., Knowledge discovery in multi-label phenotype data, () · Zbl 1009.68730
[8] B. Zhang, Z. Chen, Y.L. Murphey, Protein secondary structure prediction using machine learning, IEEE International Joint Conference on Neural Networks, July 2005.
[9] Apte, C.; Damerau, F.; Weiss, S.M., Automated learning of decision rules for text categorization, Inf. syst., 12, 3, 233-251, (1994)
[10] Natarajan, B.K., Machine learning: A theoretical approach, (1991), Morgan Kaufmann Los Alamitos, CA · Zbl 0722.68093
[11] Vapnik, V.N., Statistical learning theory, (1998), Wiley New York · Zbl 0934.62009
[12] Hsu, C.; Lin, C., A comparison of methods for multiclass support vector machines, IEEE trans. neural networks, 13, 2, 415-425, (2002)
[13] Bishop, C.M., Neural networks for pattern recognition, (1995), Oxford University Press Oxford
[14] D. Rumelhart, G. Hinton, R. Williams, Learning internal representations by error propagation, in: D.E. Rumelhart, J.L. McClelland (Eds.), Parallel Distributed Processing: Explorations in the Microstructures of Cognition, vol. 1, MIT Press, Cambridge, MA, 1986, pp. 318-362.
[15] Anthony, M.; Bartlett, P., Neural network learning: theoretical foundations, (1999), Cambridge University Press Cambridge, UK · Zbl 0968.68126
[16] Anand, R.; Mehrotra, K.; Mohan, C.K.; Ranka, S., Efficient classification for multiclass problems using modular neural networks, IEEE trans. neural networks, 6, 117-124, (1995)
[17] Gelenbe, E.; Hussain, K.F., Learning in the multiple class random neural network, IEEE trans. neural networks, 13, 6, 1257-1267, (2002)
[18] Allwein, E.L.; Schapire, R.E., Reducing multiclass to binary: a unifying approach for margin classifiers, J. Mach. learn. res., 1, 113-141, (2000) · Zbl 1013.68175
[19] Price, D.; Knerr, S., Pairwise neural network classifiers with probabilistic outputs, Neural inf. process. syst., 7, (1994)
[20] Lu, B.; Ito, M., Task decomposition and module combination based on class relations: a modular neural network for pattern classification, IEEE trans. neural networks, 10, 5, 1244-1256, (1999)
[21] Allwein, E.; Schapire, R.E.; Singer, Y., Reducing multiclass to binary: a unifying approach for margin classifiers, () · Zbl 1013.68175
[22] K. Crammer, Y. Singer, On the learnability and design of output codes for multiclass problems, in: Computational Learning Theory, 2000, pp. 35-46. · Zbl 1012.68155
[23] J. Furnkranz, Round robin classification, J. Mach. Learn. Res. (2) (2002) 721-747. · Zbl 1033.68086
[24] Hastie, T.; Tibshirani, R., Classification by pairwise coupling, (), 507-513
[25] Murphey, Y.L.; Guo, H.; Feldkamp, L.A., Neural learning from imbalanced data, Appl. intell. neural networks appl., 21, 2, 117-128, (2004) · Zbl 1075.68075
[26] Maass, W., On the computational power of winner-take-all, Neural comput., 12, 11, 2519-2536, (2000)
[27] S. Knerr, L. Personnaz, G. Dreyfus, Single-layer learning revisited: a stepwise procedure for building and training a neural network, in: Neurocomputing: Algorithms, Architectures and Applications, NATO ASI Series, Springer, Berlin, 1990.
[28] J.C. Platt, N. Cristianini, Large margin DAGs for multiclass classification, Advances in Neural Information Processing Systems, 12th ed., 2000, pp. 547-553.
[29] S. Har-Peled, D. Roth, Constraint classification: a new approach for multiclass classification and ranking, in: 13th International Conference on Algorithmic Learning Theory, 2002, pp.365-379. · Zbl 1024.68081
[30] Wu, T.-F.; Lin, C.-J.; Weng, R.C., Probability estimates for multi-class classification by pairwise coupling, J. Mach. learn. res., 5, 975-1005, (2004) · Zbl 1222.68336
[31] H. Blockeel, M. Bruynooghe, S. Dzeroski, J. Ramon, J. Struyf, Hierarchical multi-classification, in: First International Workshop on Multi-Relational Data Mining (KDD-2003), 2003.
[32] Kong, E.B.; Dietterich, T.G., Error-correcting output coding corrects bias and variance, (), 313-321
[33] Wang, K.; Zhou, S.; Liew, S.C., Building hierarchical classifiers using class proximity, (), 363-374
[34] G. James, Majority vote classifiers: theory and applications, Ph.D. Thesis, Stanford University, 1998.
[35] T.G. Dietterich, G. Bakiri, Solving multiclass learning problems via error-correcting output codes, J. Artif. Intell. Res. (2) (1995) 263-286. · Zbl 0900.68358
[36] Caruana, R., Multitask learning, Mach. learn., 28, 41-75, (1997)
[37] Berman, H.M.; Westbrook, J.; Feng, Z.; Gilliland, G.; Bhat, T.N.; Weissig, H.; Shindyalov, I.N.; Bourne, P.E., The protein data bank, Nucleic acids res., 28, 235-242, (2000)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.