×

On the stability of feature selection algorithms. (English) Zbl 1471.62267

Summary: Feature Selection is central to modern data science, from exploratory data analysis to predictive model-building. The “stability” of a feature selection algorithm refers to the robustness of its feature preferences, with respect to data sampling and to its stochastic nature. An algorithm is ‘unstable’ if a small change in data leads to large changes in the chosen feature subset. Whilst the idea is simple, quantifying this has proven more challenging – we note numerous proposals in the literature, each with different motivation and justification. We present a rigorous statistical treatment for this issue. In particular, with this work we consolidate the literature and provide (1) a deeper understanding of existing work based on a small set of properties, and (2) a clearly justified statistical approach with several novel benefits. This approach serves to identify a stability measure obeying all desirable properties, and (for the first time in the literature) allowing confidence intervals and hypothesis tests on the stability, enabling rigorous experimental comparison of feature selection algorithms.

MSC:

62F07 Statistical ranking and selection procedures
62F03 Parametric hypothesis testing
62R07 Statistical aspects of big data and data science

Software:

ElemStatLearn
PDFBibTeX XMLCite
Full Text: Link

References:

[1] Salem Alelyani. On Feature Selection Stability: A Data Perspective. PhD thesis, Arizona State University, 2013.
[2] Wilker Altidor, Taghi M. Khoshgoftaar, and Amri Napolitano. A noise-based stability evaluation of threshold-based feature selection techniques. In IEEE International Conference on Information Reuse & Integration (IRI’11), pages 240–245, 2011.
[3] Luca Baldassarre, Massimiliano Pontil, and Janaina Mouro-Miranda. Sparsity is better with stability: Combining accuracy and stability for model selection in brain decoding. Frontiers in Neuroscience, 11:62, 2017.
[4] Kenneth J Berry, Paul W Mielke, Jr, and Janis E Johnston. Permutation statistical methods: an integrated approach. Springer, 2016. · Zbl 1358.62011
[5] Anne-Laure Boulesteix and Martin Slawski. Stability and aggregation of ranked gene lists. Briefings in Bioinformatics, 10(5):556–68, 2009.
[6] Gavin Brown, Adam Pocock, Ming-Jie Zhao, and Mikel Luj´an.Conditional likelihood maximisation: a unifying framework for information theoretic feature selection. The Journal of Machine Learning Research, 13(1):27–66, 2012. · Zbl 1283.68283
[7] Chad A. Davis, Fabian Gerick, Volker Hintermair, Caroline C. Friedel, Katrin Fundel, Robert Kffner, and Ralf Zimmer. Reliable gene signatures for microarray classification: assessment of stability and performance. Bioinformatics, 22(19):2356–63, 2006.
[8] David J. Dittman, Taghi M. Khoshgoftaar, Randall Wald, and Amri Napolitano. Similarity analysis of feature ranking techniques on imbalanced dna microarray datasets. In 2012 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 1–5, 2012. 51
[9] Gregory Ditzler, Robi Polikar, and Gail Rosen. A bootstrap based neyman-pearson test for identifying variable importance. IEEE Transactions on Neural Networks and Learning Systems, 26(4):880–886, 2015.
[10] Kevin Dunne, Padraig Cunningham, and Francisco Azuaje. Solutions to instability problems with sequential wrapper-based approaches to feature selection. Technical Report TCDCS-2002-28, Trinity College Dublin, School of Computer Science, 2002.
[11] Joseph L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382, 1971.
[12] Joseph L. Fleiss, Bruce Levin, and Myunghee Cho Paik. The Measurement of Interrater Agreement, pages 598–626. John Wiley & Sons, Inc., 2004.
[13] Wilson Wen Bin Goh and Limsoon Wong. Evaluating feature-selection stability in nextgeneration proteomics. Journal of Bioinformatics and Computational Biology, 14(05): 1650029, 2016.
[14] Gokhan Gulgezen, Zehra Cataltepe, and Lei Yu. Stable and accurate feature selection. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML/PKDD), pages 455–468, 2009.
[15] Isabelle Guyon and Andr´e Elisseeff. An introduction to variable and feature selection. Journal of machine learning research, 3:1157–1182, 2003. · Zbl 1102.68556
[16] Roberto Guzm´an-Mart´ınez and Roc´ıo Alaiz-Rodr´ıguez. Feature selection stability assessment based on the jensen-shannon divergence. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML/PKDD), pages 597–612. 2011.
[17] Kilem Li Gwet. Variance estimation of nominal-scale inter-rater reliability with random selection of raters. Psychometrika, 73(3):407, 2008. · Zbl 1301.62120
[18] Yue Han and Lei Yu. A variance reduction framework for stable feature selection. Statistical Analysis and Data Mining, 5(5):428–445, 2012.
[19] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference and Prediction. Springer, 2nd edition, 2009. · Zbl 1273.62005
[20] Giuseppe Jurman, Stefano Merler, Annalisa Barla, Silvano Paoli, Antonio Galea, and Cesare Furlanello.Algebraic stability indicators for ranked lists in molecular profiling. Bioinformatics, 24(2):258–64, 2008.
[21] Alexandros Kalousis, Julien Prados, and Melanie Hilario. Stability of feature selection algorithms. In IEEE International Conference on Data Mining (ICDM’05), 2005. · Zbl 0978.68639
[22] Alexandros Kalousis, Julien Prados, and Melanie Hilario. Stability of feature selection algorithms: a study on high-dimensional spaces. Knowledge and Information Systems, 12(1):95–116, 2007. 52
[23] Iman Kamkar, Sunil Kumar Gupta, Dinh Phung, and Svetha Venkatesh. Stable feature selection with support vector machines. In Australasian Joint Conference on Artificial Intelligence (AI 2015), volume 9457 of LNCS, pages 298–308, 2015.
[24] Pavel Kr´ızek, Josef Kittler, and V´aclav Hlav´ac. Improving stability of feature selection methods. In International Conference on Computer Analysis of Images and Patterns (CAIP), volume 4673 of LNCS, pages 929–936, 2007.
[25] Ludmila I. Kuncheva. A stability index for feature selection. In Artificial Intelligence and Applications (AIAP’07), pages 390–395, 2007.
[26] Ludwig Lausser, Christoph M¨ussel, Markus Maucher, and Hans A. Kestler. Measuring and visualizing the stability of biomarker selection techniques. Computational Statistics, 28 (1):51–65, 2013. · Zbl 1305.65052
[27] Hae Woo Lee, Carl Lawton, Young Jeong Na, and Seongkyu Yoon.Robustness of chemometrics-based feature selection methods in early cancer detection and biomarker discovery. Statistical Applications in Genetics and Molecular Biology, 12(2):207–23, 2012.
[28] Jonathan L Lustgarten, Vanathi Gopalakrishnan, and Shyam Visweswaran. Measuring stability of feature selection in biomedical datasets. AMIA Annual Symposium Proceedings., pages 406–410, 2009.
[29] Nicolai Meinshausen and Peter B¨uhlmann. Stability selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(4):417–473, 2010.
[30] Sarah Nogueira. Quantifying the Stability of Feature Selection. PhD thesis, University of Manchester, 2018.
[31] Sarah Nogueira, Konstantinos Sechidis, and Gavin Brown. On the use of Spearman’s rho to measure the stability of feature rankings. In Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA), pages 381–391, 2017.
[32] Mohana Chelvan P. and Karuppasamy Perumal. A survey on feature selection stability measures. International Journal of Computer and Information Technology, 5(1):98–103, 2016.
[33] Yvan Saeys, Thomas Abeel, and Yves Van de Peer. Robust feature selection using ensemble feature selection techniques. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML/PKDD, volume 5212 of LNCS, pages 313– 325, 2008.
[34] Ahmad A. Shanab, Taghi M. Khoshgoftaar, and Randall Wald. Impact of noise and data sampling on stability of feature selection. In International Conference on Machine Learning and Applications and Workshops (ICMLA), pages 172–177, 2011.
[35] Leming Shi, Laura H. Reid, Wendell D. Jones, et al. The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements. Nature Biotechnology, 24(9):1151–61, September 2006. 53
[36] Petr Somol and Jana Novoviˇcov´a. Evaluating stability and comparing output of feature selectors that optimize feature subset cardinality. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(11):1921–1939, 2010.
[37] Randall Wald, Taghi M. Khoshgoftaar, and David J. Dittman. A new fixed-overlap partitioning algorithm for determining stability of bioinformatics gene rankers. In International Conference on Machine Learning and Applications (ICMLA), 2012a.
[38] Randall Wald, Taghi M. Khoshgoftaar, and Ahmad Abu Shanab. The effect of measurement approach and noise level on gene selection stability. In IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2012b.
[39] Randall Wald, Taghi M. Khoshgoftaar, and Amri Napolitano.Stability of filter- and wrapper-based feature subset selection. In International Conference on Tools with Artificial Intelligence. IEEE Computer Society, 2013.
[40] Søren Wichmann and David Kamholz. A stability metric for typological features. STUFLanguage Typology and Universals Sprachtypologie und Universalienforschung, 61(3):251– 262, 2008.
[41] Lei Yu, Chris H. Q. Ding, and Steven Loscalzo. Stable feature selection via dense feature groups. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2008.
[42] Lei Yu, Yue Han, and Michael E. Berens. Stable gene selection from microarray data via sample weighting. IEEE/ACM Trans. Comput. Biology Bioinform., 9(1):262–272, 2012.
[43] Min Zhang, Lin Zhang, Jinfeng Zou, Chen Yao, Hui Xiao, Qing Liu, Jing Wang, Dong Wang, Chenguang Wang, and Zheng Guo. Evaluating reproducibility of differential expression discoveries in microarray studies by considering correlated molecular changes. Bioinformatics, 2009.
[44] Ding-Xuan Zhou. On grouping effect of elastic net. Statistics & Probability Letters, 83(9): 2108 – 2112, 2013. · Zbl 1285.68145
[45] Manuela Zucknick, Sylvia Richardson, and Euan A. Stronach. Comparing the characteristics of gene expression profiles derived by univariate and multivariate classification methods. Statistical Applications in Genetics and Molecular Biology, 7(1), 2008. · Zbl 1276.92033
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.