zbMATH — the first resource for mathematics

Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants. (English) Zbl 1105.62091
Summary: This work studies the effects of sampling variability in Monte Carlo-based methods to estimate very high-dimensional systems. Recent focus in the geosciences has been on representing the atmospheric state using a probability density function, and, for extremely high-dimensional systems, various sample-based Kalman filter techniques have been developed to address the problem of real-time assimilation of system information and observations. As the employed sample sizes are typically several orders of magnitude smaller than the system dimension, such sampling techniques inevitably induce considerable variability into the state estimate, primarily through prior and posterior sample covariance matrices.
We quantify this variability with mean squared error measures for two Monte Carlo-based Kalman filter variants: the ensemble Kalman filter and the ensemble square-root Kalman filter. Expressions of the error measures are derived under weak assumptions and show that sample sizes need to grow proportionally to the square of the system dimension for bounded error growth. To reduce necessary ensemble size requirements and to address rank-deficient sample covariances, covariance-shrinking (tapering) based on the Schur product of the prior sample covariance and a positive definite function is demonstrated to be a simple, computationally feasible, and very effective technique. Rules for obtaining optimal taper functions for both stationary as well as non-stationary covariances are given, and optimal taper lengths are given in terms of the ensemble size and practical range of the forecast covariance. Results are also presented for optimal covariance inflation. The theory is verified and illustrated with extensive simulations.

62M20 Inference from stochastic processes and prediction
62H11 Directional data; spatial statistics
86A32 Geostatistics
65C05 Monte Carlo methods
15A60 Norms of matrices, numerical range, applications of functional analysis to matrix theory
Full Text: DOI
[1] ()
[2] Anderson, J.L., An ensemble adjustment Kalman filter for data assimilation, Monthly weather rev., 129, 2884-2903, (2001)
[3] J.L. Anderson, Exploring the need for localization in ensemble data assimilation using an hierarchical ensemble filter, Phys. D (2006) in press.
[4] Anderson, J.L.; Anderson, S.L., A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts, Monthly weather rev., 127, 2741-2758, (1999)
[5] Anderson, T.W., Asymtotic theory for principal component analysis, Ann. math. statist., 34, 122-148, (1963) · Zbl 0202.49504
[6] Anderson, T.W., An introduction to multivariate statistical analysis, (1984), Wiley New York · Zbl 0651.62041
[7] Arnold, S., Mathematical statistics, (1990), Prentice-Hall Englewood Cliffs, NJ · Zbl 0788.62002
[8] Bengtsson, T.; Snyder, C.; Nychka, D., Toward a nonlinear ensemble filter for high-dimensional systems, J. geophys. res., 108, 8775, (2003)
[9] P. Bickel, K. Doksum, Mathematical Statistics: Basic Ideas and Selected Topics, vol. 1, second ed., Prentice-Hall, Englewood Cliffs, NJ, 2001. · Zbl 0403.62001
[10] Bishop, C.; Etherton, B.; Majumdar, S., Adaptive sampling with the ensemble transform Kalman filter, part i: theoretical aspects, Monthly weather rev., 129, 420-436, (2001)
[11] Burgers, G.; van Leeuwen, P.J.; Evensen, G., Analysis scheme in the ensemble Kalman filter, Monthly weather rev., 126, 1719-1724, (1998)
[12] Courtier, P.; Talagrand, O., Variational assimilation of meteorological observations with the adjoint equation, part I: numerical results, Quart. J. roy. meterol. soc., 113, 1329-1347, (1987)
[13] Cressie, N.A.C., Fitting variogram models by weighted least squares, J. internat. assoc. math. geol., 17, 563-586, (1985)
[14] Cressie, N.A.C., Statistics for spatial data, (1993), Revised reprint, Wiley New York · Zbl 0468.62095
[15] Dee, D.; da Silva, A., Maximum likelihood estimation of forecast and observation error covariance parameters, part I: methodology, Monthly weather rev., 127, 1822-1834, (1999)
[16] Dee, D.; Gaspari, G.; Redder, C.; Rukhovets, L.; da Silva, A., Maximum likelihood estimation of forecast and observation error covariance parameters, part II: applications, Monthly weather rev., 127, 1835-1849, (1999)
[17] ()
[18] Evensen, G., Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics, J. geophys. res., 99, (1994), 10143-10162
[19] Evensen, G.; van Leeuwen, P.J., Assimilation of geostat altimeter data for the agulhas current using the ensemble Kalman filter, Monthly weather rev., 124, 85-96, (1996)
[20] Furrer, R., Covariance estimation under spatial dependence, J. multivariate anal., 94, 366-381, (2005) · Zbl 1066.62060
[21] Furrer, R.; Genton, M.G.; Nychka, D., Covariance tapering for interpolation of large spatial datasets, J. comput. graphical statist., 15, 502-523, (2006)
[22] Gaspari, G.; Cohn, S.E., Construction of correlation functions in two and three dimensions, Quart. J. roy. meteorol. soc., 125, 723-757, (1999)
[23] Gneiting, T., Correlation functions for atmospheric data analysis, Quart. J. roy. meteorol. soc., 125, 2449-2464, (1999)
[24] Gneiting, T., Radial positive definite functions generated by Euclid’s hat, J. multivariate anal., 69, 88-119, (1999) · Zbl 0937.60007
[25] Gneiting, T., Compactly supported correlation functions, J. multivariate anal., 83, 493-508, (2002) · Zbl 1011.60015
[26] Hamill, T.M.; Snyder, C.; Morss, R., A comparison of probabilistic forecasts from bred, singular vector and perturbed observation ensembles, Monthly weather rev., 128, 1835-1851, (2000)
[27] Hamill, T.M.; Whitaker, J.S.; Snyder, C., Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter, Monthly weather rev., 129, 2776-2790, (2001)
[28] Hollingsworth, A.; Lonnberg, P., The statistical structure of short-range forecast errors as determined from radiosonde data, part I: the wind field, Tellus, 38A, 111-136, (1986)
[29] Horn, R.A.; Johnson, C.R., Matrix analysis, (1990), Cambridge University Press Cambridge · Zbl 0704.15002
[30] Horn, R.A.; Johnson, C.R., Topics in matrix analysis, (1994), Cambridge University Press Cambridge · Zbl 0801.15001
[31] Houtekamer, P.L.; Lefaivre, L.; Derome, J.; Ritchie, H.; Mitchell, H.L., A system simulation approach to ensemble prediction, Monthly weather rev., 124, 1225-1242, (1996)
[32] Houtekamer, P.L.; Mitchell, H.L., Data assimilation using an ensemble Kalman filter technique, Monthly weather rev., 126, 796-811, (1998)
[33] Houtekamer, P.L.; Mitchell, H.L., A sequential ensemble Kalman filter for atmospheric data assimilation, Monthly weather rev., 129, 123-137, (2001)
[34] Johnstone, I.M., On the distribution of the largest eigenvalue in principal components analysis, Ann. statist., 29, 295-327, (2001) · Zbl 1016.62078
[35] Kalman, R.E., A new approach to linear filtering and prediction problems, J. basic eng., 82, 34-45, (1960)
[36] Klinker, E.; Rabier, F.; Kelly, G.; Mahfouf, J., The ECMWF operational implementation of four-dimensional variational assimilation, III: experimental results and diagnostics with operational configuration, Quart. J. roy. meterol. soc., 126, 1191-1215, (2000)
[37] Künsch, H.R., State space and hidden Markov models, (), 109-173, (Chapter 3) · Zbl 1002.62072
[38] Leeuwen, P., Comment on “data assimilation using an ensemble Kalman filter”, Monthly weather rev., 127, 1374-1379, (1999)
[39] B. Li, T. Bengtsson, P. Bickel, Curse-of-dimensionality revisited: collapse of importance sampling in very large scale systems, Technical Report #696, Department of Statistics, UC-Berkeley, 2005. · Zbl 1166.93376
[40] Liu, J.S., Monte Carlo strategies in scientific computing, (2001), Springer Series in Statistics, Springer New York · Zbl 0991.65001
[41] Lonnberg, P.; Hollingsworth, A., The statistical structure of short-range forecast errors as determined from radiosonde data, part II: the covaraince of height and wind errors, Tellus, 38A, 137-161, (1986)
[42] Lorenc, A.C., Analysis methods for numerical weather prediction, Quart. J. roy. meterol. soc., 112, 1177-1194, (1986)
[43] Mardia, K.V.; Marshall, R.J., Maximum likelihood estimation of models for residual covariance in spatial regression, Biometrika, 71, 135-146, (1984) · Zbl 0542.62079
[44] Molteni, F.; Buizza, R.; Palmer, T.N.; Petroliagis, T., The ECMWF ensemble prediction system: methodology and validation, Quart. J. roy. meteorol. soc., 122, 73-119, (1996)
[45] Pellerin, G.; Lefaivre, L.; Houtekamer, P.L.; Girard, C., Increasing the horizontal resolution of ensemble forecasts at CMC, Nonlinear process. geophys., 10, 463-468, (2003)
[46] Richardson, D., Skill and relative economic value of the ECMWF ensemble prediction system, Quart. J. roy. meteorol. soc., 126, 649-667, (2000)
[47] Stein, M.L., Minimum norm quadratic estimation of spatial variograms, J. amer. statist. assoc., 82, 765-772, (1987)
[48] Tippet, M.; Cohn, S., Adjoints and low-rank covariance representation, Nonlinear process. geophys., 8, 331-340, (2001)
[49] Tippett, M.K.; Anderson, J.L.; Bishop, C.H.; Hamill, T.M.; Whitaker, J.S., Ensemble square-root filters, Monthly weather rev., 131, 1485-1490, (2003)
[50] Toth, Z.; Kalnay, E., Ensemble forecasting at NMC: the generation of perturbations, Bull. amer. meteorol. soc., 74, 2317-2330, (1993)
[51] Toth, Z.; Kalnay, E., Ensemble forecasting at NCEP and the breeding method, Monthly weather rev., 125, 3297-3319, (1997)
[52] Tracy, C.A.; Widom, H., The distribution of the largest eigenvalue in the Gaussian ensembles, (), 461-472
[53] Wendland, H., Piecewise polynomial positive definite and compactly supported radial functions of minimal degree, Adv. comput. math., 4, 389-396, (1995) · Zbl 0838.41014
[54] Wendland, H., Error estimates for interpolation by compactly supported radial basis functions of minimal degree, J. approx. theory, 93, 258-272, (1998) · Zbl 0904.41013
[55] Whitaker, J.S.; Hamill, T.M., Ensemble data assimilation without perturbed observations, Monthly weather rev., 130, 1913-1924, (2002)
[56] Wu, Z.M., Compactly supported positive definite radial functions, Adv. comput. math., 4, 283-292, (1995) · Zbl 0837.41016
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.