×

zbMATH — the first resource for mathematics

Nonlinear multi-output regression on unknown input manifold. (English) Zbl 1386.68133
Summary: Consider unknown smooth function which maps high-dimensional inputs to multidimensional outputs and whose domain of definition is unknown low-dimensional input manifold embedded in an ambient high-dimensional input space. Given training dataset consisting of “input-output” pairs, regression on input manifold problem is to estimate the unknown function and its Jacobian matrix, as well to estimate the input manifold. By transforming high-dimensional inputs in their low-dimensional features, initial regression problem is reduced to certain regression on feature space problem. The paper presents a new geometrically motivated method for solving both interrelated regression problems.

MSC:
68T05 Learning and adaptive systems in artificial intelligence
62G08 Nonparametric regression and quantile regression
62H25 Factor analysis and principal components; correspondence analysis
68T10 Pattern recognition, speech recognition
PDF BibTeX Cite
Full Text: DOI
References:
[1] Vapnik, V.: Statistical Learning Theory. John Wiley, New York (1998) · Zbl 0935.62007
[2] Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer, New York (2009) · Zbl 1273.62005
[3] James, G., Witten, D., Hastie, T., Tibshirani, R.: An Introduction to Statistical Learning with Applications in R. Springer Texts in Statistics. Springer, New York (2013) · Zbl 1281.62147
[4] Bishop, C. M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2007) · Zbl 1107.68072
[5] Deng, L., Yu, D.: Deep Learning: Methods and Applications. NOW Publishers, Boston (2014) · Zbl 1315.68208
[6] Breiman, L, Random forests, Mach. Learn., 45, 5-32, (2001) · Zbl 1007.68152
[7] Friedman, JH, Greedy function approximation: a gradient boosting machine, Ann. Stat., 29, 1189-1232, (2001) · Zbl 1043.62034
[8] Rasmussen, C. E., Williams, C.: Gaussian Processes for Machine Learning. MIT Press, Cambridge (2006) · Zbl 1177.68165
[9] Loader, C.: Local Regression and Likelihood. Springer, New York (1999) · Zbl 0929.62046
[10] Bernstein, AV; Kuleshov, AP; Yanovich, YA; Gammerman, A (ed.); Vovk, V (ed.); Papadopoulos, H (ed.), Manifold learning in regression tasks, 414-423, (2015), Heidelberg
[11] Burnaev, E; Vovk, V, Efficiency of conformalized ridge regression, JMLR W&,CP, 35, 605-622, (2014)
[12] Burnaev, E; Panov, M; Zaytsev, A, Regression on the basis of nonstationary gaussian processes with Bayesian regularization, J. Commun. Technol. Electron., 61, 661-671, (2016)
[13] Burnaev, E; Nazarov, I, Conformalized kernel ridge regression, 45-52, (2016), USA
[14] Polyak, B. T.: Introduction to optimization. Translations Series in Mathematics and Engineering. Optimization Software Inc. Publications Division, New York (1987)
[15] Bertsekas, D. P.: Nonlinear Programming, 2nd edn. Belmont MA, Athena Scientific (1999) · Zbl 1015.90077
[16] Nesterov, Y.: Introductory Lectures on Convex Optimization. Kluwer, Boston (2004) · Zbl 1086.90045
[17] Wang, G; Gary Shan, S, Review of metamodeling techniques in support of engineering design optimization, J. Mech. Des., 129, 370-381, (2007)
[18] Forrester, A. I. J., Sobester, A., Keane, A. J.: Engineering Design via Surrogate Modelling. A Practical Guide. Wiley, New York (2008)
[19] Kuleshov, AP; Bernstein, AV, Cognitive technologies in adaptive models of complex plants, Inf. Control. Probl. Manuf., 13, 1441-1452, (2009) · Zbl 1165.62009
[20] Stone, CJ, Optimal rates of convergence for nonparametric estimators, Ann. Stat., 8, 1348-1360, (1980) · Zbl 0451.62033
[21] Stone, CJ, Optimal global rates of convergence for nonparametric regression, Ann. Stat., 10, 1040-1053, (1982) · Zbl 0511.62048
[22] Rajaram, D; Pant, RS; Rodrigues, H (ed.), An improved methodology for airfoil shape optimization using surrogate based design optimization, 147-152, (2015), London
[23] Bernstein, A; Kuleshov, A; Sviridenko, Y; Vyshinsky, V, Fast aerodynamic model for design technology, (2007)
[24] Zhu, F; Qin, N; Burnaev, EV; Bernstein, AV; Chernova, SS; Poloni, C (ed.), Comparison of three geometric parameterization methods and their effect on aerodynamic optimization, 758-772, (2011), Italy
[25] Hu, J; Tian, J; Yang, L, Functional feature embedded space mapping of fMRI data, (2006)
[26] Shen, X., Meyer, F. G.: Analysis of Event-Related fMRI Data Using Diffusion Maps. Lecture Notes in Computer Science, vol. 3565/2005 Information Processing in Medical Imaging, pp. 652-663. Springer, Berlin/Heidelberg (2005)
[27] Thirion, B; Faugeras, O, Nonlinear dimension reduction of fMRI data: the Laplacian embedding approach, 372-375, (2004), Publisher
[28] Thirion, B; Faugeras, O, Low dimensional embedding of fMRI datasets, (2008)
[29] Mannfolk, P; Wirestam, R; Nilsson, M; Ståhlberg, F; Olsrud, J, Dimensionality reduction of fMRI time series data using locally linear embedding, Magn. Reson. Mater. Phys., Biol. Med., 23, 327-338, (2010)
[30] Gerber, S., Tasdizena, T., Fletcher, P. T., Joshia, S., Whitaker, R.: On the manifold structure of the space of brain images. Lecture Notes in Computer Science, vol. 5761 Medical Image Computing and Computer-Assisted Intervention, pp. 305-312. Springer, Heidelberg (2009)
[31] Gerber, S; Tasdizena, T; Fletcher, PT; Joshia, S; Whitaker, R, Manifold modeling for brain population analysis, Med. Image Anal., 14, 643-653, (2010)
[32] Fletcher, PT, Geodesic regression on Riemannian manifolds, (2011)
[33] Fletcher, PT, Geodesic regression and the theory of least squares on Riemannian manifolds, Int. J. Comput. Vis., 105, 171-185, (2013) · Zbl 1304.62092
[34] Banerjee, M., Chakraborty, R., Ofori, E., Vaillancourt, D., Vemuri, B. C.: Nonlinear regression on Riemannian manifolds and its applications to Neuro-image analysis. Lecture Notes in Computer Science, vol. 9349 Medical Image Computing and Computer-Assisted Intervention, Part I, pp 719-727. Springer, Heidelberg (2015)
[35] Marcus, DS; Wang, TH; etal., OASIS: cross-sectional mri data in Young, middle aged, nondemented, and demented older adults, J. Cogn. Neurosci., 19, 1498-1507, (2007)
[36] Seung, HS; Lee, DD, The manifold ways of perception, Science, 290, 2268-2269, (2000)
[37] Huo, X; Ni, X; Smith, AK; Liao, T W (ed.); Triantaphyllou, E (ed.), Survey of manifold-based learning methods, 691-745, (2007), Singapore
[38] Ma, Y., Fu, Y. (eds.): Manifold Learning Theory and Applications. CRC Press, London (2011) · Zbl 0511.62048
[39] Bernstein, A. V., Kuleshov, A. P.: Low-Dimensional Data representation in data analysis. In: EI Gayar, N., Schwenker, F., Suen, C (eds.) ANNPR 2014. LNCS, vol. 8774, pp 47-58. Springer, Heidelberg (2014) · Zbl 1088.62053
[40] Kuleshov, A. P., Bernstein, A. V.: Manifold learning in data mining tasks. In: Perner, P (ed.) MLDM 2014. LNCS, vol. 8556, pp 119-133. Springer, Heidelberg (2014)
[41] Pelletier, B, Nonparametric regression estimation on closed Riemannian manifolds, J. Nonparametric Stat., 18, 57-67, (2006) · Zbl 1088.62053
[42] Loubes, J-M; Pelletier, B, A kernel-based classifier on a Riemannian manifold, Statistics and Decisions, 26, 35-51, (2008) · Zbl 1418.62161
[43] Steinke, F; Hein, M; Schölkopf, B, Nonparametric regression between general Riemannian manifolds, SIAM J. Imaging Sci., 3, 527-563, (2010) · Zbl 1195.41011
[44] Levina, E., Bickel, P. J.: Maximum likelihood estimation of intrinsic dimension. In: Saul, L., Weiss, Y., Bottou, L (eds.) Advances in Neural Information Processing Systems, vol. 17, pp 777-784. MIT Press, Cambridge (2005)
[45] Fan, M; Qiao, H; Zhang, B, Intrinsic dimension estimation of manifolds by incising balls, Pattern Recogn., 42, 780-787, (2009) · Zbl 1162.68405
[46] Fan, M; Gu, N; Qiao, H; Zhang, B, Intrinsic dimension estimation of data by principal component analysis, 1-8, (2010)
[47] Rozza, A; Lombardi, G; Rosa, M; Casiraghi, E; Campadelli, P; Maino, G (ed.); Foresti, G (ed.), IDEA: intrinsic dimension estimation algorithm, 433-442, (2011), Heidelberg
[48] Campadelli, P., Casiraghi, E., Ceruti, C., Rozza, A.: Intrinsic Dimension Estimation: Relevant Techniques and a Benchmark Framework. Mathematical Problems in Engineering vol. 2015, Article ID 759567, pp. 1-21 (2015) · Zbl 1395.68244
[49] Banerjee, M; Chakraborty, R; Ofori, E; Okun, MS; Vaillancourt, D; Vemuri, BC, A nonlinear regression technique for manifold valued data with applications to medical image analysis, (2016)
[50] Hinkle, J; Muralidharan, P; Fletcher, PT, Polynomial regression on Riemannian manifolds, (2012)
[51] Liu, G; Lin, Z; Yu, Y, Multi-output regression on the output manifold, Pattern Recogn., 42, 2737-2743, (2009) · Zbl 1175.68376
[52] Shi, X; Styner, M; Lieberman, J; Ibrahim, JG; Lin, W; Zhu, H, Intrinsic regression models for manifold-valued data, J. Amer. Stat. Assoc., 5762, 192-199, (2009)
[53] Kim, HJ; Bendlin, BB; etal., Multivariate general linear models (MGLM) on Riemannian manifolds with applications to statistical analysis of diffusion weighted images, (2014)
[54] Kim, H. J., Adluru, N., et al.: Canonical Correlation analysis on Riemannian Manifolds and its Applications. Lecture Notes in Computer Science, vol. 8690 Computer Vision - ECCV 2014, pp. 251-267. Springer, Heidelberg (2014)
[55] Bickel, P., Li, B.: Local polynomial regression on unknown manifolds. IMS Lecture notes - Monograph Series, vol. 54 Complex Datasets and Inverse Problems: Tomography Networks and Beyond, pp. 177-186 (2007)
[56] Aswani, A; Bickel, P; Tomlin, C, Regression on manifolds: estimation of the exterior derivative, Ann. Stat., 39, 48-81, (2011) · Zbl 1209.62063
[57] Cheng, M-Y; Wu, H-T, Local linear regression on manifolds and its geometric interpretation, J. Am. Stat. Assoc., 108, 1421-1434, (2013) · Zbl 1426.62402
[58] Yang, Y; Dunson, DB, Bayesian manifold regression, (2014)
[59] Einbeck, J; Evers, L, Localized regression on principal manifolds, (2010)
[60] Niyogi, P; Smale, S; Weinberger, S, Finding the homology of submanifolds with high confidence from random samples, Discrete Comput. Geom., 39, 419-441, (2008) · Zbl 1148.68048
[61] Pennec, X, Probabilities and statistics on Riemannian manifolds: basic tools for geometric measurements, (1999)
[62] Yanovich, YU, Asymptotic properties of local sampling on manifolds, J. Math. Stat., 12, 157-175, (2016)
[63] Adragni, KP; Cook, RD, Sufficient dimension reduction and prediction in regression. phil, Philos. Trans. R. Soc. A Math. Phys. Eng. Sci., 367, 4385-4405, (2009) · Zbl 1185.62109
[64] Lee, K-Y; Li, B; Chiaromonte, F, A general theory for nonlinear sufficient dimension reduction: formulation and estimation, Ann. Stat., 41, 221-249, (2013) · Zbl 1347.62018
[65] Chen, J; Deng, S-J; Huo, X, Electricity price curve modeling and forecasting by manifold learning, IEEE Trans. Power Syst., 23, 877-888, (2008)
[66] Lee, JA; Verleysen, M; Saeys, Y (ed.), Quality assessment of dimensionality reduction based on k-ary neighborhoods, 21-35, (2008), Belgium
[67] Lee, JA; Verleysen, M, Quality assessment of dimensionality reduction: rank-based criteria, Neurocomputing, 72, 1431-1443, (2009)
[68] Bernstein, AV; Kuleshov, AP, Data-based manifold reconstruction via tangent bundle manifold learning, (2014)
[69] Freedman, D, Efficient simplicial reconstructions of manifold from their samples, IEEE Trans. Pattern Anal. Mach. Intell., 24, 1349-1357, (2002)
[70] Levin, D.: Mesh-independent surface interpolation. In: Hamann, Brunnett, Mueller (eds.) Mathematics and Visualization Series, Geometric Modeling for Scientific Visualization, part 1, pp 37-49. Springer, Berlin, Heidelberg (2003)
[71] Kolluri, R., Shewchuk, J. R., O’Brien, J. F.: Spectral surface reconstruction from noisy point clouds Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (SGP’04), pp 11-21. ACM Press, New York (2004)
[72] Boissonnat, J-D; Ghosh, A, Manifold reconstruction using tangential Delaunay complexes, Discrete Comput. Geom., 51, 221-267, (2014) · Zbl 1312.68209
[73] Karygianni, S; Frossard, P, Tangent-based manifold approximation with locally linear models, Signal Process., 104, 232-247, (2014)
[74] Canas, G. D., Poggio, T., Rosasco, L.: Learning manifolds with K-Means and K-Flats arXiv:1209.1121v4 [cs.LG], 19 Feb 2013, pp. 1-19 (2013) · Zbl 1418.62161
[75] Jollie, T.: Principal component analysis. Springer, New York (2002)
[76] Kramer, M, Nonlinear principal component analysis using autoassociative neural networks, AIChE J, 37, 233-243, (1991)
[77] Hinton, GE; Salakhutdinov, RR, Reducing the dimensionality of data with neural networks, Science, 313, 504-507, (2006) · Zbl 1226.68083
[78] Saul, LK; Roweis, ST, Nonlinear dimensionality reduction by locally linear embedding, Science, 290, 2323-2326, (2000)
[79] Saul, LK; Roweis, ST, Think globally, fit locally: unsupervised learning of low dimensional manifolds, J. Mach. Learn. Res., 4, 119-155, (2003) · Zbl 1093.68089
[80] Zhang, Z; Zha, H, Principal manifolds and nonlinear dimension reduction via local tangent space alignment, SIAM J. Sci. Comput., 26, 313-338, (2005) · Zbl 1077.65042
[81] Bernstein, A. V., Kuleshov, A. P.: Tangent bundle manifold learning via Grassmann&Stiefel eigenmaps arXiv:1212.6031v1 [cs.LG], December 2012, pp. 1-25 (2012)
[82] Kuleshov, A., Bernstein, A., Yanovich, Y.U.: Asymptotically optimal method in Manifold estimation. In: Márkus, L., Prokaj, V. (eds.) Abstracts of the 29th European Meeting of Statisticians (EMS-2003), Budapest, Hungary, 20-25 July 2013, p. 325 (2013) · Zbl 1226.68083
[83] Law, M. H. C., Jain, A. K.: Nonlinear manifold learning for data stream. In: Berry, M., Dayal, U., Kamath, C., Skillicorn, D (eds.) Proceedings of the 4 \^{}{th} SIAM International Conference on Data Mining, Like Buena Vista, Florida, USA, 33-44 (2004) · Zbl 1077.65042
[84] Law, MHC; Jain, AK, Incremental nonlinear dimensionality reduction by manifold learning, IEEE Trans. Pattern Anal. Mach. Intell., 28, 377-391, (2006)
[85] Gao, X; Jiye Liang, J, An improved incremental nonlinear dimensionality reduction for isometric data embedding, Inf. Process. Lett., 115, 492-501, (2015) · Zbl 1312.68166
[86] Kouropteva, O., Okun, O., Pietikäinen, M.: Incremental locally linear embedding algorithm. In: Kalviainen, H. et al. (eds.) 14th Scandinavian Conference on Image Analysis (SCIA) 2005, LNCS, vol. 3540, pp 521-530. Springer-Verlag, Berlin, Heidelberg (2005)
[87] Kouropteva, O; Okun, O; Pietikäinen, M, Incremental locally linear embedding, Pattern Recogn., 38, 1764-1767, (2005) · Zbl 1077.68800
[88] Schuon, S., Ðurković, M., Diepold, K., Scheuerle, J., Markward, S.: Truly Incremental Locally Linear Embedding Proceedings of the CoTeSys 1st International Workshop on Cognition for Technical Systems (6-8 October 2008), munich, Germany, 5 pp (2008)
[89] Jia, P; Yin, J; Huang, X; Hu, D, Incremental Laplacian eigenmaps by preserving adjacent information between data points, Pattern Recogn. Lett., 30, 1457-1463, (2009)
[90] Liu, X., Yin, J., Feng, Z., Dong, J.: Incremental Manifold Learning Via Tangent Space Alignment. Schwenker, F., Marinai, S. (eds.) ANNPR 2006, LNAI, Vol. 4087, pp. 107-121. Springer-Verlag. Berlin, Heidelberg (2006) · Zbl 1007.68152
[91] Abdel-Mannan, O., Ben Hamza, A., Youssef, A.: Incremental line tangent space alignment algorithm Proceedings of 2007 Canadian Conference on Electrical and Computer Engineering (CCECE 2007), 22-26 April 2007, Vancouver, pp. 1329-1332. IEEE (2007)
[92] Han, Z; Meng, D-Y; Xu, Z-B; Gu, N-N, Incremental alignment manifold learning, J. Comput. Sci. Technol., 26, 153-165, (2011)
[93] Chao Tan, C., Guan, J.: A new manifold learning algorithm based on incremental spectral decomposition. In: Zhou, S., Zhang, S., Karypis, G (eds.) 8th International Conference on Advanced Data Mining and Applications (ADMA) Nanjing, China, December 15-18, 2012, LNCS, vol. 7713, pp 149-160. Springer-Verlag, Berlin, Heidelberg (2012) · Zbl 1043.62034
[94] Kuleshov, A., Bernstein, A.: Incremental construction of low-dimensional data representations. In: Schwenker, F., EI Gayar, N., Abbas, H. M., Trentin, E (eds.) ANNPR 2016. LNCS, vol. 9896, pp 55- 67. Springer, Heidelberg (2016)
[95] Golub, G. H., Van Loan, C. F.: Matrix Computation, 3rd edn. Johns Hopkins University Press, Baltimore, MD (1996)
[96] Bernstein, AV; Kuleshov, AP, Manifold learning: generalizing ability and tangent proximity, Int. J. Softw. Eng. Inform., 7, 359-390, (2013)
[97] Jost, J.: Riemannian Geometry and Geometric Analysis, 6th edn. Springer-Verlag, Berlin, Heidelberg (2011) · Zbl 1227.53001
[98] Lee, J. M.: Manifolds and Differential Geometry Graduate Studies in Mathematics, vol. 107. American Mathematical Society, Providence (2009)
[99] Perrault-Joncas, D., Meilă, M.: Non-linear dimensionality reduction: riemannian metric estimation and the problem of geometric recovery arXiv:1305.7255v1 [stat.ML], 30 May 2013, pp. 1-25 (2013)
[100] Kuleshov, A. P., Bernstein, A. V.: Extended Regression on Manifolds Estimation. Lecture Notes in Artificial Intelligence, vol. 9653 Conformal and Probabilistic Prediction with Applications, pp. 208-228. Springer, Heidelberg (2016)
[101] Singer, A; Wu, H-T, Vector diffusion maps and the connection Laplacian, Commun. Pure. Appl. Math., 65, 1067-1144, (2012) · Zbl 1320.68146
[102] Tyagi, H., Vural, E., Frossard, P.: Tangent space estimation for smooth embeddings of Riemannian manifold arXiv:1208.1065v2 [stat.CO], 17 May 2013, pp. 1-35 (2013)
[103] Kaslovsky, DN; Meyer, FG, Non-asymptotic analysis of tangent space perturbation, Inf. Inf. J. IMA, 3, 134-187, (2014) · Zbl 1309.62106
[104] Hamm, J.: Lee, Daniel D.: Grassmann discriminant analysis: a unifying view on subspace-based learning Proceedings of the 25 \^{}{th} International Conference on Machine Learning (ICML 2008), pp. 376-383 (2008)
[105] Wolf, L; Shashua, A, Learning over sets using kernel principal angles, J. Mach. Learn. Res., 4, 913-931, (2003) · Zbl 1098.68679
[106] Cormen, T., Leiserson, C., Rivest, R., Stein, C.: Introduction to Algorithms. MIT Press, Cambridge (2001) · Zbl 1047.68161
[107] Fredman, M; Tarjan, RE, Fibonacci heaps and their uses in improved network optimization algorithms, J. Assoc. Comput. Mach., 34, 596-615, (1987) · Zbl 1412.68048
[108] Bernstein, A. V., Kuleshov, A. P., Yanovich, Y.A.: Information preserving and locally isometric&conformal embedding via tangent manifold learning Proceedings of the International IEEE Conference on Data Science and Advanced Analytics (DSAA 2015), pp 1-9. IEEE Computer Society, Piscataway, USA (2015)
[109] Trefethen, L.N., Bau, D. III: Numerical. Linear Algebra. SIAM, Philadelphia (1997) · Zbl 0874.65013
[110] Wasserman, L.: All of Nonparametric Statistics. Springer Texts in Statistics, Berlin (2007) · Zbl 1099.62029
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.