QUADRO: a supervised dimension reduction method via Rayleigh quotient optimization. (English) Zbl 1317.62054

Summary: We propose a novel Rayleigh quotient based sparse quadratic dimension reduction method – named QUADRO (Quadratic Dimension Reduction via Rayleigh Optimization) – for analyzing high-dimensional data. Unlike in the linear setting where Rayleigh quotient optimization coincides with classification, these two problems are very different under nonlinear settings. In this paper, we clarify this difference and show that Rayleigh quotient optimization may be of independent scientific interests. One major challenge of Rayleigh quotient optimization is that the variance of quadratic statistics involves all fourth cross-moments of predictors, which are infeasible to compute for high-dimensional applications and may accumulate too many stochastic errors. This issue is resolved by considering a family of elliptical models. Moreover, for heavy-tail distributions, robust estimates of mean vectors and covariance matrices are employed to guarantee uniform convergence in estimating nonpolynomially many parameters, even though only the fourth moments are assumed. Methodologically, QUADRO is based on elliptical models which allow us to formulate the Rayleigh quotient maximization as a convex optimization problem. Computationally, we propose an efficient linearized augmented Lagrangian method to solve the constrained optimization problem. Theoretically, we provide explicit rates of convergence in terms of Rayleigh quotient under both Gaussian and general elliptical models. Thorough numerical results on both synthetic and real datasets are also provided to back up our theoretical results.


62H30 Classification and discrimination; cluster analysis (statistical aspects)
62G20 Asymptotic properties of nonparametric inference
Full Text: DOI arXiv Euclid


[1] Bickel, P. J., Ritov, Y. and Tsybakov, A. B. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705-1732. · Zbl 1173.62022
[2] Bindea, G., Mlecnik, B., Hackl, H., Charoentong, P., Tosolini, M., Kirilovsky, A., Fridman, W.-H., Pagès, F., Trajanoski, Z. and Galon, J. (2009). ClueGO: A cytoscape plug-in to decipher functionally grouped gene ontology and pathway annotation networks. Bioinformatics 25 1091-1093.
[3] Cai, T. and Liu, W. (2011). A direct estimation approach to sparse linear discriminant analysis. J. Amer. Statist. Assoc. 106 1566-1577. · Zbl 1233.62129
[4] Cai, T., Liu, W. and Luo, X. (2011). A constrained \(\ell_{1}\) minimization approach to sparse precision matrix estimation. J. Amer. Statist. Assoc. 106 594-607. · Zbl 1232.62087
[5] Catoni, O. (2012). Challenging the empirical mean and empirical variance: A deviation study. Ann. Inst. Henri Poincaré Probab. Stat. 48 1148-1185. · Zbl 1282.62070
[6] Chen, X., Zou, C. and Cook, R. D. (2010). Coordinate-independent sparse sufficient dimension reduction and variable selection. Ann. Statist. 38 3696-3723. · Zbl 1204.62107
[7] Cook, R. D. and Weisberg, S. (1991). Comment on “Sliced inverse regression for dimension reduction.” J. Amer. Statist. Assoc. 86 328-332. · Zbl 0742.62044
[8] Coudret, R., Liquet, B. and Saracco, J. (2014). Comparison of sliced inverse regression approaches for underdetermined cases. J. SFdS 155 72-96. · Zbl 1316.62068
[9] Fan, J. and Fan, Y. (2008). High-dimensional classification using features annealed independence rules. Ann. Statist. 36 2605-2637. · Zbl 1360.62327
[10] Fan, J., Feng, Y. and Tong, X. (2012). A road to classification in high dimensional space. J. Roy. Statist. Soc. B 74 745-771.
[11] Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348-1360. · Zbl 1073.62547
[12] Fan, J., Xue, L. and Zou, H. (2014). Strong oracle optimality of folded concave penalized estimation. Ann. Statist. 42 819-849. · Zbl 1305.62252
[13] Fan, J., Ke, Z. T., Liu, H. and Xia, L. (2015). Supplement to “QUADRO: A supervised dimension reduction method via Rayleigh quotient optimization.” . · Zbl 1317.62054
[14] Feller, W. (1966). An Introduction to Probability Theory and Its Applications. Vol. II . Wiley, New York. · Zbl 0138.10207
[15] Fisher, R. A. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics 7 179-188.
[16] Friedman, J. H. (1989). Regularized discriminant analysis. J. Amer. Statist. Assoc. 84 165-175.
[17] Guo, Y., Hastie, T. and Tibshirani, R. (2005). Regularized discriminant analysis and its application in microarrays. Biostatistics 1 1-18. · Zbl 1170.62382
[18] Han, F. and Liu, H. (2012). Transelliptical component analysis. Adv. Neural Inf. Process. Syst. 25 368-376.
[19] Han, F., Zhao, T. and Liu, H. (2013). CODA: High dimensional copula discriminant analysis. J. Mach. Learn. Res. 14 629-671. · Zbl 1320.62145
[20] Jiang, B. and Liu, J. S. (2013). Sliced inverse regression with variable selection and interaction detection. Preprint. Available at . arXiv:1304.4056
[21] Kendall, M. G. (1938). A new measure of rank correlation. Biometrika 30 81-93. · Zbl 0019.13001
[22] Kent, J. T. (1991). Discussion of Li (1991). J. Amer. Statist. Assoc. 86 336-337.
[23] Li, K.-C. (1991). Sliced inverse regression for dimension reduction. J. Amer. Statist. Assoc. 86 316-342. · Zbl 0742.62044
[24] Li, K.-C. (2000). High dimensional data analysis via the SIR/PHD approach. Lecture notes, Dept. Statistics, UCLA, Los Angeles, CA. Available at .
[25] Li, B. and Wang, S. (2007). On directional regression for dimension reduction. J. Amer. Statist. Assoc. 102 997-1008. · Zbl 1469.62300
[26] Li, L. and Yin, X. (2008). Sliced inverse regression with regularizations. Biometrics 64 124-131. · Zbl 1139.62055
[27] Liu, H., Han, F., Yuan, M., Lafferty, J. and Wasserman, L. (2012). High-dimensional semiparametric Gaussian copula graphical models. Ann. Statist. 40 2293-2326. · Zbl 1297.62073
[28] Luparello, C. (2013). Aspects of collagen changes in breast cancer. J. Carcinogene Mutagene S 13:007. .
[29] Maruyama, Y. and Seo, T. (2003). Estimation of moment parameter in elliptical distributions. J. Japan Statist. Soc. 33 215-229. · Zbl 1067.62064
[30] Shao, J., Wang, Y., Deng, X. and Wang, S. (2011). Sparse linear discriminant analysis by thresholding for high dimensional data. Ann. Statist. 39 1241-1265. · Zbl 1215.62062
[31] Wei, Z. and Li, H. (2007). A Markov random field model for network-based analysis of genomic data. Bioinformatics 23 1537-1544.
[32] Witten, D. M. and Tibshirani, R. (2011). Penalized classification using Fisher’s linear discriminant. J. R. Stat. Soc. Ser. B. Stat. Methodol. 73 753-772. · Zbl 1228.62079
[33] Wu, H.-M. (2008). Kernel sliced inverse regression with applications to classification. J. Comput. Graph. Statist. 17 590-610.
[34] Zhao, T., Roeder, K. and Liu, H. (2013). Positive semidefinite rank-based correlation matrix estimation with application to semiparametric graph estimation. Unpublished manuscript.
[35] Zhao, P. and Yu, B. (2006). On model selection consistency of Lasso. J. Mach. Learn. Res. 7 2541-2563. · Zbl 1222.62008
[36] Zhong, W., Zeng, P., Ma, P., Liu, J. S. and Zhu, Y. (2005). RSIR: Regularized sliced inverse regression for motif discovery. Bioinformatics 21 4169-4175.
[37] Zou, H. (2006). The adaptive lasso and its oracle properties. J. Amer. Statist. Assoc. 101 1418-1429. · Zbl 1171.62326
[38] Zou, H. and Li, R. (2008). One-step sparse estimates in nonconcave penalized likelihood models. Ann. Statist. 36 1509-1533. · Zbl 1142.62027
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.