×

Cascade interpolation learning with double subspaces and confidence disturbance for imbalanced problems. (English) Zbl 1434.68470

Summary: In this paper, a new ensemble framework named Cascade Interpolation Learning with Double subspaces and Confidence disturbance (CILDC) is designed for the imbalanced classification problems. Developed from the Cascade Forest of the Deep Forest which is the stacking based tree ensembles for big data issues with less hyper-parameters, CILDC aims to generalize the cascade model for more base classifiers. Specifically, CILDC integrates base classifiers through the double subspaces strategy and the random under-sampling preprocessing. Further, one simple but effective confidence disturbance technique is introduced to CILDC to tune the threshold deviation for imbalanced samples. In detail, the disturbance coefficients are multiplied to various confidence vectors before interpolating in each level of CILDC, and the ideal threshold can be adaptively learned through the cascade structure. Furthermore, both the Random Forest and the Naive Bayes are suitable to be the base classifier for CILDC. Subsequently, comprehensive comparison experiments on typical imbalanced datasets demonstrate both the effectiveness and generalization of CILDC.

MSC:

68T05 Learning and adaptive systems in artificial intelligence

Software:

XGBoost; SMOTEBoost; KEEL
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Bach, F. R.; Heckerman, D.; Horvitz, E., Considering cost asymmetry in learning classifiers, Journal of Machine Learning Research (JMLR), 7, 4, 1713-1741 (2006) · Zbl 1222.68137
[2] Breiman, L., Bagging predictors, Machine Learning, 24, 123-140 (1996) · Zbl 0858.68080
[3] Breiman, L., Random forests, Machine Learning, 45, 5-32 (2001) · Zbl 1007.68152
[4] Bryll, R.; Gutierrez-Osuna, R.; Quek, F., Attribute bagging: Improving accuracy of classifier ensembles by using random feature subsets, Pattern Recognition, 36, 6, 1291-1302 (2003) · Zbl 1033.68092
[5] Chawla, N. V.; Japkowicz, N.; Kotcz, A., Editorial: Special issue on learning from imbalanced data sets, ACM Sigkdd Explorations Newsletter, 6, 1-6 (2004)
[6] Chawla, N.; Lazarevic, A.; Hall, L.; Bowyer, K., Smoteboost: Improving prediction of the minority class in boosting, Knowledge Discovery in Databases: PKDD, 2003, 107-119 (2003)
[7] Chen, T.; Guestrin, C., Xgboost: A scalable tree boosting system, (Proceedings of the 22nd ACM sigkdd international conference on knowledge discovery and data mining (2016), ACM), 785-794
[8] Demšar, J., Statistical comparisons of classifiers over multiple data sets, Journal of Machine Learning Research (JMLR), 7, 1, 1-30 (2006) · Zbl 1222.68184
[9] Donoho, D. L.; Tanner, J., Precise undersampling theorems, Proceedings of the IEEE, 98, 6, 913-924 (2010)
[10] Džeroski, S.; Ženko, B., Is combining classifiers with stacking better than selecting the best one?, Machine Learning, 54, 3, 255-273 (2004) · Zbl 1101.68077
[11] Estabrooks, A.; Jo, T.; Japkowicz, N., A multiple resampling method for learning from imbalanced data sets, Computational Intelligence, 20, 1, 18-36 (2004)
[12] Fernández-Delgado, M.; Cernadas, E.; Barro, S.; Amorim, D., Do we need hundreds of classifiers to solve real world classification problems?, Journal of Machine Learning Research (JMLR), 15, 1, 3133-3181 (2014) · Zbl 1319.62005
[13] Freund, Y.; Schapire, R. E., A desicion-theoretic generalization of on-line learning and an application to boosting, (European conference on computational learning theory (1995), Springer), 23-37
[14] Friedman, M., The use of ranks to avoid the assumption of normality implicit in the analysis of variance, Journal of the American Statistical Association, 32, 200, 675-701 (1937) · JFM 63.1098.02
[15] Friedman, N.; Dan, G.; Goldszmidt, M., Bayesian network classifiers, Machine Learning, 29, 2-3, 131-163 (1997) · Zbl 0892.68077
[16] Galar, M.; Fernandez, A.; Barrenechea, E.; Bustince, H.; Herrera, F., A review on ensembles for the class imbalance problem: Bagging-, boosting-, and hybrid-based approaches, IEEE Transactions on Systems Man and Cybernetics Part C Applications and Reviews, 42, 4, 463-484 (2012)
[17] Gao, D.; Ding, J.; Zhu, C., Integrated fisher linear discriminants: An empirical study, Pattern Recognition, 47, 2, 789-805 (2014) · Zbl 1326.68219
[18] He, H.; Garcia, E. A., Learning from imbalanced data, IEEE Transactions on Knowledge and Data Engineering, 21, 9, 1263-1284 (2009)
[19] Ho, T. K., The random subspace method for constructing decision forests, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20, 8, 832-844 (1998)
[20] Huang, G., Liu, Z., Weinberger, K. Q., & Laurens, V. D. M. (2016). Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition; Huang, G., Liu, Z., Weinberger, K. Q., & Laurens, V. D. M. (2016). Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition
[21] Hutchinson, B.; Deng, L.; Yu, D., Tensor deep stacking networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 35, 8, 1944-1957 (2013)
[22] Krawczyk, B., Learning from imbalanced data: Open challenges and future directions, Progress in Artificial Intelligence, 5, 4, 221-232 (2016)
[23] Liu, F. T.; Kai, M. T.; Yu, Y.; Zhou, Z. H., Spectrum of variable-random trees, Journal of Artificial Intelligence Research, 32, 355-384 (2008) · Zbl 1182.68255
[24] Lu, Y.; Cheung, Y.-m.; Tang, Y. Y., Hybrid sampling with bagging for class imbalance learning, (Pacific-Asia conference on knowledge discovery and data mining (2016), Springer), 14-26
[25] Olshen, R.; Breiman, L.; Friedman, J.; Stone, C. J., Classification and regression tree (1984), Chapman and Hall · Zbl 0541.62042
[26] Palangi, H., Deng, L., & Ward, R. K. (2014). Recurrent deep-stacking networks for sequence classification. In IEEE China summit and international conference on signal and information processing; Palangi, H., Deng, L., & Ward, R. K. (2014). Recurrent deep-stacking networks for sequence classification. In IEEE China summit and international conference on signal and information processing
[27] Ramentol, E.; Caballero, Y.; Bello, R.; Herrera, F., Smote-rsb*: A hybrid preprocessing approach based on oversampling and undersampling for high imbalanced data-sets using smote and rough sets theory, Knowledge and Information Systems, 33, 2, 245-265 (2012)
[28] Seiffert, C.; Khoshgoftaar, T. M.; Van Hulse, J.; Napolitano, A., Rusboost: A hybrid approach to alleviating class imbalance, IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans, 40, 1, 185-197 (2010)
[29] Sun, Z.; Song, Q.; Zhu, X.; Sun, H.; Xu, B., A novel ensemble method for classifying imbalanced data, Pattern Recognition, 48, 5, 1623-1637 (2015)
[30] Tao, D.; Tang, X.; Li, X.; Wu, X., Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, 28, 7, 1088-1099 (2006)
[31] Tian, T., Ji, W., & Gao, D. (2015). Threshold optimization of pseudo-inverse linear discriminants based on overall accuracies. In International joint conference on neural networks; Tian, T., Ji, W., & Gao, D. (2015). Threshold optimization of pseudo-inverse linear discriminants based on overall accuracies. In International joint conference on neural networks
[32] Ting, K. M.; Witten, I. H., Issues in stacked generalization, JAIR, 10, 271-289 (1999) · Zbl 0915.68075
[33] Triguero, I.; González, S.; Moyano, J. M.; García, S.; Alcalá-Fdez, J., Keel 30: An open source software for multi-stage analysis in data mining, International Journal of Computational Intelligence Systems, 10, 1238-1249 (2017)
[34] Van Hulse, J.; Khoshgoftaar, T. M.; Napolitano, A., Experimental perspectives on learning from imbalanced data, (Proceedings of the 24th international conference on Machine learning (2007), ACM), 935-942
[35] Wang, S.; Yao, X., Diversity analysis on imbalanced data sets by using ensemble models, (Computational Intelligence and Data Mining, 2009 CIDM’09. Computational Intelligence and Data Mining, 2009 CIDM’09, IEEE symposium on (2009), IEEE), 324-331
[36] Wolpert, D. H., Stacked generalization, Neural Networks, 5, 2, 241-259 (1992)
[37] Yu, Z.; Chen, H.; Liu, J.; You, J.; Leung, H., Hybrid \(k\)-nearest neighbor classifier, IEEE Transactions on Cybernetics, 46, 6, 1263-1275 (2016)
[38] Yu, Z.; Li, L.; Liu, J.; Han, G., Hybrid adaptive classifier ensemble, IEEE Transactions on Cybernetics, 45, 2, 177-190 (2015)
[39] Yu, Z., Lu, Y., Zhang, J., You, J., & Wong, H.-S., et al. Progressive semisupervised learning of multiple classifiers, IEEE Transactions on Cybernetics; Yu, Z., Lu, Y., Zhang, J., You, J., & Wong, H.-S., et al. Progressive semisupervised learning of multiple classifiers, IEEE Transactions on Cybernetics
[40] Yu, Z.; Wang, Z.; You, J.; Zhang, J.; Liu, J., A new kind of nonparametric test for statistical comparison of multiple classifiers over multiple datasets, IEEE Transactions on Cybernetics, 47, 12, 4418-4431 (2017)
[41] Yu, Z., Wang, D., Zhao, Z., Chen, C. P., & You, J., et al. Hybrid incremental ensemble learning for noisy real-world data classification, IEEE Transactions on Cybernetics; Yu, Z., Wang, D., Zhao, Z., Chen, C. P., & You, J., et al. Hybrid incremental ensemble learning for noisy real-world data classification, IEEE Transactions on Cybernetics
[42] Zhou, Z. H., Cost-sensitive learning, (International conference on modeling decisions for artificial intelligence (2011), Springer), 17-18
[43] Zhou, Z. H., A general boosting procedure, (Ensemble methods: Foundations and algorithms (2012)), 23-24
[44] Zhou, Z. H., & Feng, J. (2017). Deep forest: Towards an alternative to deep neural networks. In Proceedings of the twenty-sixth international joint conference on artificial intelligence; Zhou, Z. H., & Feng, J. (2017). Deep forest: Towards an alternative to deep neural networks. In Proceedings of the twenty-sixth international joint conference on artificial intelligence
[45] Zhu, T.; Lin, Y.; Liu, Y., Synthetic minority oversampling technique for multiclass imbalance problems, Pattern Recognition, 72, 327-340 (2017)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.