×

High-order accurate solution of the incompressible Navier-Stokes equations on massively parallel computers. (English) Zbl 1391.76476

Summary: The emergence of “petascale” supercomputers requires us to develop today’s simulation codes for (incompressible) flows by codes which are using numerical schemes and methods that are better able to exploit the offered computational power. In that spirit, we present a massively parallel high-order Navier-Stokes solver for large incompressible flow problems in three dimensions. The governing equations are discretized with finite differences in space and a semi-implicit time integration scheme. This discretization leads to a large linear system of equations which is solved with a cascade of iterative solvers. The iterative solver for the pressure uses a highly efficient commutation-based preconditioner which is robust with respect to grid stretching. The efficiency of the implementation is further enhanced by carefully setting the (adaptive) termination criteria for the different iterative solvers. The computational work is distributed to different processing units by a geometric data decomposition in all three dimensions. This decomposition scheme ensures a low communication overhead and excellent scaling capabilities. The discretization is thoroughly validated. First, we verify the convergence orders of the spatial and temporal discretizations for a forced channel flow. Second, we analyze the iterative solution technique by investigating the absolute accuracy of the implementation with respect to the different termination criteria. Third, Orr-Sommerfeld and Squire eigenmodes for plane Poiseuille flow are simulated and compared to analytical results. Fourth, the practical applicability of the implementation is tested for transitional and turbulent channel flow. The results are compared to solutions from a pseudospectral solver. Subsequently, the performance of the commutation-based preconditioner for the pressure iteration is demonstrated. Finally, the excellent parallel scalability of the proposed method is demonstrated with a weak and a strong scaling test on up to \(O(10^4)\) processing units and \(O(10^{11})\) grid points.

MSC:

76M20 Finite difference methods applied to problems in fluid mechanics
65M06 Finite difference methods for initial value and initial-boundary value problems involving PDEs
65Y05 Parallel numerical computation

Software:

BoomerAMG
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Adams, M.; Brezina, M.; Hu, J.; Tuminaro, R., Parallel multigrid smoothing: polynomial versus Gauss-Seidel, J. Comput. Phys., 188, 593-610 (2003) · Zbl 1022.65030
[2] Armfield, S. W., Finite difference solutions of the Navier-Stokes equations on staggered and non-staggered grids, Comput. Fluids, 20, 1, 1-17 (1991) · Zbl 0731.76044
[3] G.A. Blaisdell, N.N. Mansour, W.C. Reynolds, Numerical simulation of compressible homogeneous turbulence, Technical Report Rep. TF-50, Department of Mechanical Engineering, Stanford University, 1991.; G.A. Blaisdell, N.N. Mansour, W.C. Reynolds, Numerical simulation of compressible homogeneous turbulence, Technical Report Rep. TF-50, Department of Mechanical Engineering, Stanford University, 1991.
[4] Brandt, A.; Dinar, N., Multi-grid solutions to elliptic flow problems, (Parter, S., Numerical Methods for Partial Differential Equations (1979), Academic Press: Academic Press New York), 53-147
[5] Brüger, A.; Gustafsson, B.; Lötstedt, P.; Nilsson, J., High order accurate solution of the incompressible Navier-Stokes equations, J. Comput. Phys., 203, 49-71 (2005) · Zbl 1061.76040
[6] Chen, K., Matrix Preconditioning Techniques and Applications (2005), Cambridge University Press: Cambridge University Press Cambridge · Zbl 1079.65057
[7] Choi, H.; Moin, P., Effects of the computational time step on numerical solutions of turbulent flow, J. Comput. Phys., 113, 1-4 (1994) · Zbl 0807.76051
[8] D.A. Donzis, P.K. Yeung, D. Pekurovsky, Turbulence simulations on \(\mathcal{O} 10^4\); D.A. Donzis, P.K. Yeung, D. Pekurovsky, Turbulence simulations on \(\mathcal{O} 10^4\)
[9] Eidson, T. M.; Erlebacher, G., Implementation of a fully balanced periodic tridiagonal solver on a parallel distributed memory architecture, Concurrency Comput.: Pract. Exp., 7, 4, 273-302 (1995)
[10] Elman, H.; Howle, V. E.; Shadid, J.; Shuttleworth, R.; Tuminaro, R., A taxonomy and comparison of parallel block multi-level preconditioners for the incompressible Navier-Stokes equations, J. Comput. Phys., 227, 3, 1790-1808 (2008) · Zbl 1290.76023
[11] Elman, H. C., Preconditioning for the steady-state Navier-Stokes equations with low viscosity, SIAM J. Sci. Comput., 20, 1299-1316 (1999) · Zbl 0935.76057
[12] Fujimoto, T.; Ranade, R. R., Two characterization of inverse-positive matrices: the Hawkins-Simon condition and the Le Chatelier-Braun principle, Electron. J. Linear Algebra, 11, 59-65 (2004) · Zbl 1069.15020
[13] Greenbaum, A., Iterative Methods for Solving Linear Systems (1997), SIAM: SIAM Philadelphia · Zbl 0883.65022
[14] Hackbusch, W., Multi-Grid Methods and Applications (1985), Springer: Springer Berlin · Zbl 0585.65030
[15] R. Henniger, L. Kleiser, Simulation of gravity-driven flows using an iterative high-order accurate Navier-Stokes solver, in: Direct and Large-Eddy Simulations VII, Trieste, 2008.; R. Henniger, L. Kleiser, Simulation of gravity-driven flows using an iterative high-order accurate Navier-Stokes solver, in: Direct and Large-Eddy Simulations VII, Trieste, 2008.
[16] R. Henniger, L. Kleiser, E. Meiburg, Direct numerical simulation of a model estuary, in: Sixth Int. Symp. on Turb. Shear Flow Phenom., Seoul, 2009.; R. Henniger, L. Kleiser, E. Meiburg, Direct numerical simulation of a model estuary, in: Sixth Int. Symp. on Turb. Shear Flow Phenom., Seoul, 2009. · Zbl 1273.86009
[17] Henson, V. E.; Yang, U. M., BoomerAMG: a parallel algebraic multigrid solver and preconditioner, Appl. Numer. Math., 41, 155-177 (2000) · Zbl 0995.65128
[18] Hess, R.; Joppich, W., A comparison of parallel multigrid and a fast Fourier transform algorithm for the solution of the Helmholtz equation in numerical weather prediction, Parallel Comput., 22, 1503-1512 (1997) · Zbl 0906.68205
[19] Hoyas, S.; Jimenez, J., Scaling of the velocity fluctuations in turbulent channels up to \(Re_\tau = 2003\), Phys. Fluids, 18, 011702 (2006)
[20] Jeong, J.; Hussain, F., On the identification of a vortex, J. Fluid Mech., 285, 69-94 (1994) · Zbl 0847.76007
[21] Karniadakis, G. E.; Israeli, M.; Orszag, S. A., High-order splitting methods for incompressible Navier-Stokes equations, J. Comput. Phys., 97, 414-443 (1991) · Zbl 0738.76050
[22] Kleiser, L.; Zang, T. A., Numerical simulation of transition in wall-bounded shear flows, Annu. Rev. Fluid Mech., 23, 495-537 (1991)
[23] Le Borne, S., Hierarchical matrix preconditioners for the Oseen equations, Comput. Vis. Sci., 11, 3, 147-157 (2006)
[24] Le Borne, S., Block computation and representation of a sparse nullspace basis of a rectangular matrix, Linear Algebra Appl., 428, 11-12, 2455-2467 (2008) · Zbl 1142.65041
[25] Lele, S. K., Compact finite difference schemes with spectral-like resolution, J. Comput. Phys., 103, 16-42 (1992) · Zbl 0759.65006
[26] LeVeque, R. J.; Li, Z., The immersed interface method for elliptic equations with discontinuous coefficients and singular sources, SIAM J. Numer. Anal., 31, 4, 1019-1044 (1994) · Zbl 0811.65083
[27] Li, Y., Wavenumber-extended high-order upwind-biased finite-difference schemes for convective scalar transport, J. Comput. Phys., 133, 235-255 (1997) · Zbl 0878.65071
[28] A. Lundbladh, D.S. Henningson, A.V. Johansson, An efficient spectral integration method for the solution of the Navier-Stokes equations, Technical Report FFA TN 1992-28, Aeronautical Research Institute of Sweden, FFA, 1992.; A. Lundbladh, D.S. Henningson, A.V. Johansson, An efficient spectral integration method for the solution of the Navier-Stokes equations, Technical Report FFA TN 1992-28, Aeronautical Research Institute of Sweden, FFA, 1992.
[29] Matheson, L. R.; Tarjan, E., Analysis of multigrid algorithms on massively parallel computers: architectural implications, J. Parallel Distrib. Comput., 33, 33-43 (1996)
[30] Mattor, N.; Williams, T. J.; Hewett, D. W., Algorithm for solving tridiagonal matrix problems in parallel, Parallel Comput., 21, 1769-1782 (1995)
[31] Moin, P.; Kim, J., On the numerical solution of time-dependent viscous incompressible fluid flows involving solid boundaries, J. Comput. Phys., 35, 381-392 (1980) · Zbl 0425.76027
[32] Moin, P.; Mahesh, K., Direct numerical simulation: a tool in turbulence research, Annu. Rev. Fluid Mech., 30, 539-578 (1998) · Zbl 1398.76073
[33] Moriya, K.; Nodera, T., Breakdown-free ML(k)BiCGStab algorithm for non-hermitian linear systems, (Gervasi, O.; etal., Computational Science and its Applications. Computational Science and its Applications, Lecture Notes in Computer Science, vol. 3483 (2005), Springer), 978-988
[34] Patankar, S. V.; Spalding, D. B., A calculation procedure for heat, mass and momentum transfer in three-dimensional parabolic flows, Int. J. Heat Mass Transfer, 15, 10, 1787-1806 (1972) · Zbl 0246.76080
[35] Peskin, C. S., The immersed boundary method, Acta Numer., 11, 1-39 (2002)
[36] Povitsky, A., Parallelization of pipelined algorithms for sets of linear banded systems, J. Parallel Distrib. Comput., 59, 68-97 (1999)
[37] P. Schlatter, Large-eddy simulation of transition and turbulence in wall-bounded shear flow. PhD thesis, ETH Zürich, Zürich, Switzerland, 2005. Diss. ETH No. 16000.; P. Schlatter, Large-eddy simulation of transition and turbulence in wall-bounded shear flow. PhD thesis, ETH Zürich, Zürich, Switzerland, 2005. Diss. ETH No. 16000.
[38] Simens, M. P.; Jimenez, J.; Hoyas, S.; Mizuno, Y., A high-resolution code for turbulent boundary layers, J. Comput. Phys., 228, 4218-4231 (2009) · Zbl 1273.76009
[39] Spalart, P. R.; Moser, R. D.; Rogers, M. M., Spectral methods for the Navier-Stokes equations with one infinite and two periodic directions, J. Comput. Phys., 96, 297-324 (1991) · Zbl 0726.76074
[40] Swarztrauber, P. N.; Hammond, S. W., A comparison of optimal FFTs on torus and hypercube multicomputers, Parallel Comput., 27, 847-859 (2001) · Zbl 0972.68003
[41] Turek, S., A comparative study of time-stepping techniques for the incompressible Navier-Stokes equations, Int. J. Numer. Methods Fluids, 22, 10, 987-1011 (1996) · Zbl 0864.76052
[42] van der, H. A., Vorst. BiCGSTAB: a fast and smoothly converging variant of Bi-CG for the solution of nonsymmetric linear systems, SIAM J. Sci. Comput., 13, 631-644 (1992) · Zbl 0761.65023
[43] Wesseling, J., Principles of Computational Fluid Dynamics (2001), Springer: Springer Berlin
[44] Wittum, G., Multi-grid methods for Stokes and Navier-Stokes equations—transforming smoothers: algorithm and numerical results, Numer. Math., 54, 543-563 (1989) · Zbl 0645.76031
[45] A.A. Wray, Very low storage time-advancement schemes, Technical report, NASA Ames Research Center, 1986.; A.A. Wray, Very low storage time-advancement schemes, Technical report, NASA Ames Research Center, 1986.
[46] Xu, J., Benchmarks on tera-scalable models for DNS of turbulent channel flow, Parallel Comput., 33, 12, 780-794 (2007)
[47] Zhang, F., The Schur Complement and its Applications (2005), Springer: Springer New York · Zbl 1075.15002
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.