×

Ensemble transform algorithms for nonlinear smoothing problems. (English) Zbl 07149722

Summary: Several numerical tools designed to overcome the challenges of smoothing in a nonlinear and non-Gaussian setting are investigated for a class of particle smoothers. The considered family of smoothers is induced by the class of linear ensemble transform filters which contains classical filters such as the stochastic ensemble Kalman filter, the ensemble square root filter, and the recently introduced nonlinear ensemble transform filter. Further the ensemble transform particle smoother is introduced and particularly highlighted as it is consistent in the particle limit and does not require assumptions with respect to the family of the posterior distribution. The linear update pattern of the considered class of linear ensemble transform smoothers allows one to implement important supplementary techniques such as adaptive spread corrections, hybrid formulations, and localization in order to facilitate their application to complex estimation problems. These additional features are derived and numerically investigated for a sequence of increasingly challenging test problems.

MSC:

65C05 Monte Carlo methods
62M20 Inference from stochastic processes and prediction
93E11 Filtering in stochastic control theory
62F15 Bayesian inference
86A22 Inverse problems in geophysics

Software:

EnKF
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] W. Acevedo, J. de Wiljes, and S. Reich, Second-order accurate ensemble transform particle filters, SIAM J. Sci. Comput., 39 (2017), pp. 1834-1850. · Zbl 1373.65001
[2] M. Arulampalam, S. Maskell, N. Gordon, and T. Clapp, A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking, IEEE Trans. Signal Process., 50 (2002), pp. 174-188, https://doi.org/10.1109/78.978374.
[3] C. Bishop and D. Hodyss, Adaptive ensemble covariance localization in ensemble \(4\) d-var state estimation, Mon. Weather Rev., 139 (2011), pp. 1241-1255.
[4] M. Bocquet, Localization and the iterative ensemble Kalman smoother, Quart. J. Roy. Meteorol. Soc., 142 (2016), pp. 1075-1089, https://doi.org/10.1002/qj.2711.
[5] M. Bocquet and P. Sakov, An iterative ensemble Kalman smoother, Quart. J. Roy. Meteorol. Soc., 140 (2014), pp. 1521-1535.
[6] M. Bonavita, L. Torrisi, and F. Marcucci, The ensemble Kalman filter in an operational regional NWP system: Preliminary results with real observations, Quart. J. Roy. Meteorol. Soc., 134 (2008), pp. 1733-1744, https://doi.org/10.1002/qj.313.
[7] A. Carrassi, M. Bocquet, L. Bertino, and G. Evensen, Data assimilation in the geosciences: An overview of methods, issues, and perspectives, WIREs, Climate Change, 9 (2018), pp. 1-50.
[8] E. P. Chassignet and J. Verron, eds., Ocean Weather Forecasting: An Integrated View of Oceanography, Springer, Dordrecht, The Netherlands, 2006.
[9] Y. Chen and D. S. Oliver, Localization and regularization for iterative ensemble smoothers, Comput. Geosci., 21 (2017), pp. 12-30. · Zbl 1387.86042
[10] Y. Chen and S. Reich, Assimilating data into scientific models: An optimal coupling perspective, in Nonlinear Data Assimilation, Front. Appl. Dyn. Syst. Rev. Tutor. 2, Springer, New York, 2015, pp. 75-118. · Zbl 1330.62004
[11] N. Chustagulprom, S. Reich, and M. Reinhardt, A hybrid ensemble transform particle filter for nonlinear and spatially extended dynamical systems, SIAM/ASA J. Uncertain. Quantif., 4 (2016), pp. 592-608. · Zbl 1343.65009
[12] T. C. Clapp and S. J. Godsill, Fixed-lag smoothing using sequential importance sampling, in Bayesian Statistics 6, Oxford University Press, Oxford, 1999, pp. 743-752. · Zbl 1179.62110
[13] E. Cosme, J.-M. Brankart, J. Verron, P. Brasseur, and M. Krysta, Implementation of a reduced rank square-root smoother for high resolution ocean data assimilation, Ocean Model., 33 (2010), pp. 87-100.
[14] E. Cosme, J. M. Brankart, J. Verron, P. Brasseur, and M. Krysta, Implementation of a reduced rank square-root smoother for high resolution ocean data assimilation, Ocean Model., 33 (2010), pp. 87-100, https://doi.org/10.1016/j.ocemod.2009.12.004.
[15] M. Cuturi, Sinkhorn distances: Lightspeed computation of optimal transport, in NIPS 2013, 2013 Curran Associates, Red Hook, NY, pp. 2292-2300.
[16] F. L. Dimet and O. Talagrand, Variational algorithms for analysis and assimilation of meteorological observations: Theoretical aspects, Tellus, 38A (1986), pp. 97-110.
[17] A. Doucet, N. de Freitas, and N. Gordon, eds., Sequential Monte Carlo Methods in Practice, Springer, Berlin, 2001. · Zbl 0967.00022
[18] A. Doucet, S. J. Godsill, and C. Andrieu, On sequential Monte Carlo sampling methods for Bayesian filtering, Statist. Comput., 10 (2000), pp. 197-208.
[19] G. Evensen, Data Assimilation. The Ensemble Kalman Filter, Springer, New York, 2006. · Zbl 1157.86001
[20] G. Evensen, Analysis of iterative ensemble smoothers for solving inverse problems, Comput. Geosci., 22 (2018), pp. 885-908. · Zbl 1405.86018
[21] G. Evensen and P. J. van Leeuwen, An ensemble Kalman smoother for nonlinear dynamics, Mon. Weather Rev., 128 (2000), pp. 1852-1867.
[22] D. Fairbairn, Comparison of the Ensemble Transform Kalman Filter with the Ensemble Transform Kalman Smoother, Master’s thesis, University of Reading, Reading, England, 2009.
[23] A. Farchi and M. Bocquet, Review article: Comparison of local particle filters and new implementations, Nonlinear Processes Geophys., 25 (2018), pp. 765-807.
[24] M. Fisher, M. Leutbecher, and G. A. Kelly, On the equivalence between Kalman smoothing and weak-constraint four-dimensional variational data assimilation, Quart. J. Roy. Meteorol. Soc., 131 (2005), pp. 3235-3246.
[25] M. Frei and H. Künsch, Bridging the ensemble Kalman and particle filters, Biometrika, 100 (2013), pp. 781-800. · Zbl 1279.62199
[26] G. Gaspari and S. Cohn, Construction of correlation functions in two and three dimensions, Quart. J. Roy. Meteorol. Soc., 125 (1999), pp. 723-757.
[27] S. J. Godsill, A. Doucet, and M. West, Monte Carlo smoothing for nonlinear time series, J. Amer. Statist. Assoc., 99 (2012), pp. 156-168. · Zbl 1089.62517
[28] T. Hamill, J. Whitaker, and C. Snyder, Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter, Mon. Weather. Rev., 129 (2001), pp. 2776-2790.
[29] H. Hersbach, Decomposition of the continuous ranked probability score for ensemble prediction systems, Weather Forecast., 15 (2000), pp. 559-570.
[30] P. Houtekamer and H. Mitchell, A sequential ensemble Kalman filter for atmospheric data assimilation, Mon. Weather Rev., 129 (2001), pp. 123-136.
[31] B. Hunt, E. Kostelich, and I. Szunyogh, Efficient data assimilation for spatialtemporal chaos: A local ensemble transform Kalman filter, Phys. D, 230 (2007), pp. 112-137. · Zbl 1115.62030
[32] A. Jazwinski, Stochastic Processes and Filtering Theory, Academic Press, New York, 1970. · Zbl 0203.50101
[33] R. E. Kalman, A new approach to linear filtering and prediction problems, Trans. ASME J. Basic Eng., 82 (1960), pp. —45.
[34] P. Kirchgessner, J. Tödter, B. Ahrens, and L. Nerger, The smoother extension of the nonlinear ensemble transform filter, Tellus A, 69 (2017), 1327766.
[35] G. Kitagawa, Monte Carlo filter and smoother for non-Gaussian nonlinear state space models, J. Comput. Graph. Statist., 5 (1996), pp. 1-25.
[36] G. Kitagawa and S. Sato, Monte Carlo smoothing and self-organising state-space model, in Practice, in Sequential Monte Carlo Methods, Springer, New York, 2001, pp. 177-195. · Zbl 1056.93581
[37] E. Lorenz, Deterministic non-periodic flows, J. Atmos. Sci., 20 (1963), pp. 130-141. · Zbl 1417.37129
[38] E. Lorenz, Predictibility: A problem partly solved, in Proceedings of a Seminar on Predictibility, Weather and Forecasting, Vol. 1, ECMWF, Reading, UK, 1996, pp. 1-18.
[39] M. C. Mackey and L. Glass, Oscillation and chaos in physiological control systems, Science, 197 (1977), pp. 287-289, https://doi.org/10.1126/science.267326. · Zbl 1383.92036
[40] R. McCann, Existence and uniqueness of monotone measure-preserving maps, Duke Math. J., 80 (1995), pp. 309-323. · Zbl 0873.28009
[41] T. Miyoshi and M. Kunii, The local ensemble transform Kalman filter with the weather research and forecasting model: Experiments with real observations, Pure Appl. Geophys., 169 (2011), pp. 321-333, https://doi.org/10.1007/s00024-011-0373-4.
[42] M. Morzfeld, D. Hodyss, and J. Poterjoy, Variational particle smoothers and their localization, Quart. J. Roy. Meteorol. Soc., 144 (2018), pp. 806-825.
[43] M. Morzfeld, D. Hodyss, and C. Snyder, What the collapse of the ensemble Kalman filter tells us about particle filters, Tellus A, 69 (2017).
[44] L. Nerger, S. Schulte, and A. Bunse-Gerstner, On the influence of model nonlinearity and localization on ensemble Kalman smoothing, Quart. J. Roy. Meteorol. Soc., 140 (2014), pp. 2249-2259, https://doi.org/10.1002/qj.2293.
[45] M. Nurujjaman, A. Apte, and P. Vinayachandran, Data assimilation using ensemble transform Kalman filter (ETKF) in ROMS model for Indian Ocean, Eur. Phys. J. Spec. Top., 222 (2013), pp. 875-883, https://doi.org/10.1140/epjst/e2013-01890-3.
[46] O. Pele and M. Werman, Fast and robust earth mover’s distances, in IEEE 12th International Conference on Computer Vision, IEEE, Piscataway, NJ, 2009, pp. 460-467.
[47] J. Poterjoy, A localized particle filter for high-dimensional nonlinear systems, Mon. Weather Rev., 144 (2015), pp. 59-76 https://doi.org/10.1175/MWR-D-15-0163.1.
[48] R. Potthast, A. Walter, and A. Rhodin, A localised adaptive particle filter within an operational NWP framework, Mon. Weather Rev., 147 (2019), pp. 345-362.
[49] S. Reich and C. Cotter, Probabilistic Forecasting and Bayesian Data Assimilation, Cambridge University Press, Cambridge, 2015. · Zbl 1314.62005
[50] S. Reich and C. J. Cotter, Ensemble filter techniques for intermittent data assimilation, in Large Scale Inverse Problems. Computational Methods and Applications in the Earth Sciences, M. Cullen, F. M. A., S. Kindermann, and R. Scheichl, eds., Radon Ser. Comput. Appl. Math. 13, Walter de Gruyter, Berlin, 2013, pp. 91-134. · Zbl 1291.65032
[51] S. Robert and H. R. Künsch, Localizing the ensemble Kalman particle filter, Tellus A, 69 (2017), 1282016, https://doi.org/10.1080/16000870.2017.1282016.
[52] S. Hu, Q. Zhang, J. Wang, and Z. Chen, Real-time particle filtering and smoothing algorithms for detecting abrupt changes in neural ensemble spike activity, J Neurophysiol, 119 (2018), pp. 1394-1410.
[53] S. Särkkä, Bayesian Filtering and Smoothing, Cambridge University Press, Cambridge, 2013. · Zbl 1274.62021
[54] F. G. Shuman, Numerical methods in weather prediction: II. Smoothing and filtering, Mon. Weather Rev., 85 (1957), pp. 357-361.
[55] C. Snyder, T. Bengtsson, P. Bickel, and J. Anderson, Obstacles to high-dimensional particle filtering, Mon. Weather Rev., 136 (2008), pp. 4629-4640, https://doi.org/10.1175/2008MWR2529.1.
[56] C. Snyder, T. Bengtsson, and M. Morzfeld, Performance bounds for particle filters using the optimal proposal, Mon. Weather Rev., 143 (2015), pp. 4750-4761, https://doi.org/10.1175/MWR-D-15-0144.1.
[57] M. Tippett, J. Anderson, G. Bishop, T. Hamill, and J. Whitaker, Ensemble square root filters, Mon. Weather Rev., 131 (2003), pp. 1485-1490.
[58] J. Tödter and B. Ahrens, A second-order exact ensemble square root filter for nonlinear data assimilation, Mon. Weather Rev., 143 (2015), pp. 1347-1367.
[59] C. Villani, Topics in Optimal Transportation, American Mathematical Society, Providence, RI, 2003. · Zbl 1106.90001
[60] C. Villani, Optimal Transportation: Old and New, Springer, Berlin, 2009. · Zbl 1156.53003
[61] B. Weir, R. N. Miller, and Y. H. Spitz, A potential implicit particle method for high-dimensional systems, Nonlinear Processes Geophys., 20 (2013), pp. 1047-1060.
[62] J. S. Whitaker and G. P. Compo, An ensemble Kalman smoother for reanalysis, in Symposium on Observations, Data Assimilation, and Probabilistic Prediction, American Meterological Society, Boston, MA, 2002, pp. 144-147.
[63] F. D. Y Petetin, A particle smoothing implementation of the fully-adapted auxiliary particle filter: An alternative to auxiliary particle filters, IEEE Statistical Signal Processing Workshop (SSP), IEEE, Piscataway, NJ, 2011, pp. 217-220.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.