×

Efficient parallelization for volume-coupled multiphysics simulations on hierarchical Cartesian grids. (English) Zbl 1441.76084

Summary: One of the main challenges for multiphysics simulations of volume-coupled problems is data exchange, which becomes a serious bottleneck for large-scale applications executed on distributed memory computer architectures. Unlike surface coupling, the transfer of volumetric information via network communication can be too inefficient. To circumvent this problem, a new concept based on a joint hierarchical mesh is proposed, where the coupled solvers use different cells of a single shared grid. A domain decomposition method based on space-filling curves guarantees that all coupling data can be exchanged locally by in-memory operations. Furthermore, the interleaved execution of the solvers ensures that there is no additional overhead due to communication barriers. To demonstrate the efficiency of the presented method for realistic three-dimensional problems, it is used to predict the acoustic field of a turbulent jet in a direct-hybrid simulation, in which fluid dynamics and aeroacoustics are directly coupled. The results confirm the good parallel scalability of the approach and show the design of the coupling algorithm to be more efficient than a standard hybrid flow-aeroacoustics scheme coupled via disk I/O.

MSC:

76M20 Finite difference methods applied to problems in fluid mechanics
65N55 Multigrid methods; domain decomposition for boundary value problems involving PDEs
65Y05 Parallel numerical computation
76Q05 Hydro- and aero-acoustics
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Liu, Y.; Dowling, A. P.; Swaminathan, N.; Dunstan, T. D., Spatial correlation of heat release rate and sound emission from turbulent premixed flames, Combust. Flame, 159, 7, 2430-2440 (2012)
[2] Schlimpert, S.; Koh, S. R.; Pausch, K.; Meinke, M.; Schröder, W., Analysis of combustion noise of a turbulent premixed slot jet flame, Combust. Flame, 175, 292-306 (2017)
[3] Lafortune, P.; Arís, R.; Vázquez, M.; Houzeaux, G., Coupled electromechanical model of the heart: Parallel finite element formulation, Int. J. Numer. Methods Biomed. Eng., 28, 1, 72-86 (2012) · Zbl 1242.92015
[4] Xia, H.; Wong, K.; Zhao, X., A fully coupled model for electromechanics of the heart, Comput. Math. Methods Med., 2012, 1-10 (2012) · Zbl 1256.92013
[5] Sugiyama, T.; Kusano, K., Multi-scale plasma simulation by the interlocking of magnetohydrodynamic model and particle-in-cell kinetic model, J. Comput. Phys., 227, 2, 1340-1352 (2007) · Zbl 1203.76127
[6] Daldorff, L. K.; Tóth, G.; Gombosi, T. I.; Lapenta, G.; Amaya, J.; Markidis, S.; Brackbill, J. U., Two-way coupling of a global Hall magnetohydrodynamics model with a local implicit particle-in-cell model, J. Comput. Phys., 268, Supplement C, 236-254 (2014) · Zbl 1349.82140
[7] Makwana, K.; Keppens, R.; Lapenta, G., Two-way coupling of magnetohydrodynamic simulations with embedded particle-in-cell simulations, Comput. Phys. Commun. (2017)
[8] Ren, W.; E., W., Heterogeneous multiscale method for the modeling of complex fluids and micro-fluidics, J. Comput. Phys., 204, 1, 1-26 (2005) · Zbl 1143.76541
[9] Yasuda, S.; Yamamoto, R., A model for hybrid simulations of molecular dynamics and computational fluid dynamics, Phys. Fluids, 20, 11, 113101 (2008) · Zbl 1182.76847
[10] Losito, R.; Marque, S., Coupled analysis of electromagnetic, thermo-mechanical effects on RF accelerating structures, (Proceedings of the European Particle Accelerator Conference 2002 (2002)), 2166-2168
[11] Kenjeres̆, S.; Hanjalić, K.; Renaudier, S.; Stefani, F.; Gerbeth, G.; Gailitis, A., Coupled fluid-flow and magnetic-field simulation of the Riga dynamo experiment, Phys. Plasmas, 13, 12, 122308 (2006)
[12] Klocke, F.; Zeis, M.; Harst, S.; Klink, A.; Veselovac, D.; Baumgrtner, M., Modeling and simulation of the electrochemical machining (ECM) material removal process for the manufacture of aero engine components, Procedia CIRP, 8, Supplement C, 265-270 (2013), 14th CIRP Conference on Modeling of Machining Operations (CIRP CMMO)
[13] Hackert-Oschätzchen, M.; Paul, R.; Kowalick, M.; Martin, A.; Meichsner, G.; Schubert, A., Multiphysics simulation of the material removal in jet electrochemical machining, Procedia CIRP, 31, Supplement C, 197-202 (2015), 15th CIRP Conference on Modelling of Machining Operations (15th CMMO)
[14] Sussman, M.; Puckett, E. G., A coupled level set and volume-of-fluid method for computing 3D and axisymmetric incompressible two-phase flows, J. Comput. Phys., 162, 2, 301-337 (2000) · Zbl 0977.76071
[15] Marella, S.; Krishnan, S.; Liu, H.; Udaykumar, H. S., Sharp interface Cartesian grid method i: An easily implemented technique for 3D moving boundary computations, J. Comput. Phys., 210, 1, 1-31 (2005) · Zbl 1154.76359
[16] Hu, X. Y.; Khoo, B. C.; Adams, N. A.; Huang, F. L., A conservative interface method for compressible flows, J. Comput. Phys., 219, 2, 553-578 (2006) · Zbl 1102.76038
[17] Schneiders, L.; Günther, C.; Meinke, M.; Schröder, W., An efficient conservative cut-cell method for rigid bodies interacting with viscous compressible flows, J. Comput. Phys., 311, 62-86 (2016) · Zbl 1349.74345
[18] Bogey, C.; Bailly, C.; Juvé, D., Computation of flow noise using source terms in linearized Euler’s equations, AIAA J., 40, 2, 235-243 (2002)
[19] Manoha, E.; Herrero, C.; Khelil, S. B.; Guillen, P.; Sagaut, P.; Mary, I., Numerical prediction of airfoil aerodynamic noise, (8th AIAA/CEAS Aeroacoustics Conference & Exhibit (2002), American Institute of Aeronautics and Astronautics)
[20] Koh, S. R.; Schröder, W.; Meinke, M., Turbulence and heat excited noise sources in single and coaxial jets, J. Sound Vib., 329, 786-803 (2010)
[21] Bungartz, H.-J.; Lindner, F.; Gatzhammer, B.; Mehl, M.; Scheufele, K.; Shukaev, A.; Uekermann, B., PreCICE a fully parallel library for multi-physics surface coupling, Comput. Fluids, 141, Supplement C, 250-258 (2016), Advances in Fluid-Structure Interaction · Zbl 1390.76004
[22] Krause, R.; Zulian, P., A parallel approach to the variational transfer of discrete fields between arbitrarily distributed unstructured finite element meshes, SIAM J. Sci. Comput., 38, 3, C307-C333 (2016) · Zbl 06601529
[23] Bakhtiari, A.; Malhotra, D.; Raoofy, A.; Mehl, M.; Bungartz, H.-J.; Biros, G., A parallel arbitrary-order accurate AMR algorithm for the scalar advection-diffusion equation, (Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, no. 44. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, no. 44, SC ’16 (2016), IEEE Press: IEEE Press Piscataway, NJ, USA)
[24] Schlüter, J.; Wu, X.; van der Weide, E.; Hahn, S.; Alonso, J.; Pitsch, H., Multi-code simulations: A generalized coupling approach, 17th AIAA Computational Fluid Dynamics Conference. 17th AIAA Computational Fluid Dynamics Conference, AIAA Pap., 2005-4997 (2005)
[25] Joppich, W.; Kürschner, M., MpCCI - A tool for the simulation of coupled applications, Concurr. Comput., 18, 2, 183-192 (2005)
[26] Gatzhammer, B.; Mehl, M.; Neckel, T., A coupling environment for partitioned multiphysics simulations applied to fluid-structure interaction scenarios, Procedia Comput. Sci., 1, 1, 681-689 (2010)
[27] Bungartz, H.-J.; Benk, J.; Gatzhammer, B.; Mehl, M.; Neckel, T., Partitioned simulation of fluid-structure interaction on Cartesian grids, (Fluid Structure Interaction II (2010), Springer Science +Business Media), 255-284 · Zbl 1213.74110
[28] Jaure, S.; Duchaine, F.; Staffelbach, G.; Gicquel, L. Y.M., Massively parallel conjugate heat transfer methods relying on large eddy simulation applied to an aeronautical combustor, Comput. Sci. Discov., 6, 1 (2013)
[29] Schlottke-Lakemper, M.; Yu, H.; Sven Berger; Meinke, M.; Schröder, W., A fully coupled hybrid computational aeroacoustics method on hierarchical Cartesian meshes, Comput. Fluids, 144, 137-153 (2017) · Zbl 1390.76830
[30] Kornhaas, M.; Sternel, D. C.; Schäfer, M., Efficiency investigation of a parallel hierarchical grid based aeroacoustic code for low Mach numbers and complex geometries, (Pereira, J. C.F.; Sequeira, A., V European Conference on Computational Fluid Dynamics, ECCOMAS CFD 2010 (2010))
[31] Kornhaas, M.; Schäfer, M.; Sternel, D. C., Efficient numerical simulation of aeroacoustics for low Mach number flows interacting with structures, Comput. Mech., 55, 6, 1143-1154 (2015) · Zbl 1325.76136
[32] Müthing, S., A flexible framework for multi physics and multi domain PDE simulations (2015), Universität Stuttgart
[33] DeZeeuw, D.; Powell, K. G., An adaptively refined Cartesian mesh solver for the euler equations, J. Comput. Phys., 104, 1, 56-68 (1993) · Zbl 0766.76066
[34] Lintermann, A.; Schlimpert, S.; Grimmen, J. H.; Günther, C.; Meinke, M.; Schröder, W., Massively parallel grid generation on HPC systems, Comput. Methods Appl. Mech. Eng., 277, 131-153 (2014) · Zbl 1423.76366
[35] Günther, C.; Meinke, M.; Schröder, W., A flexible level-set approach for tracking multiple interacting interfaces in embedded boundary methods, Comput. Fluids, 102, 182-202 (2014)
[36] Sagan, H., (Space-Filling Curves. Space-Filling Curves, Universitext (1994), Springer New York) · Zbl 0806.01019
[37] Houwen, P. V.D., Construction of Integration Formulae for Initial Value Problems (North-Holland Series in Applied Mathematics and Mechanics; Vol. 19) (1976), Elsevier Science Publishing Co Inc.,U.S.
[38] Zennaro, M., Natural continuous extensions of Runge-Kutta methods, Math. Comput., 46, 173 (1986), 119-119 · Zbl 0608.65043
[39] Enright, W. H.; Jackson, K. R.; Nrsett, S. P.; Thomsen, P. G., Interpolants for Runge-Kutta formulas, ACM Trans. Math. Softw., 12, 3, 193-218 (1987) · Zbl 0617.65068
[40] Geiser, G.; Marinc, D.; Schröder, W., Comparison of source reconstruction methods for hybrid aeroacoustic predictions, Int. J. Aeroacoust., 12, 7-8, 639-662 (2014)
[41] Ewert, R.; Schröder, W., Acoustic perturbation equations based on flow decomposition via source filtering, J. Comput. Phys., 188, 365-398 (2003) · Zbl 1022.76050
[42] van Leer, B., Towards the ultimate conservative difference scheme. V. A second-order sequel to Godunov’s method, J. Comput. Phys., 32, 1, 101-136 (1979) · Zbl 1364.65223
[43] Liou, M.-S.; Steffen, C. J., A new flux splitting scheme, J. Comput. Phys., 107, 1, 23-39 (1993) · Zbl 0779.76056
[44] Meinke, M.; Schröder, W.; Krause, E.; Rister, T., A comparison of second- and sixth-order methods for large-eddy simulations, Comput. Fluids, 31, 695-718 (2002) · Zbl 1027.76025
[45] Boris, J. P.; Grinstein, F. F.; Oran, E. S.; Kolbe, R. L., New insights into large eddy simulation, Fluid Dyn. Res., 10, 4-6, 199-228 (1992)
[46] Hartmann, D.; Meinke, M.; Schröder, W., An adaptive multilevel multigrid formulation for Cartesian hierarchical grid methods, Comput. Fluids, 37, 1103-1125 (2008) · Zbl 1237.76085
[47] Pogorelov, A.; Meinke, M.; Schröder, W., Cut-cell method based large-eddy simulation of tip-leakage flow, Phys. Fluids, 27, 7 (2015)
[48] Hu, F. Q.; Hussaini, M. Y.; Patrick Rasetarinera, An analysis of the discontinuous Galerkin method for wave propagation problems, J. Comput. Phys., 151, 921-946 (1999) · Zbl 0933.65113
[49] Kopriva, D.; Woodruff, S.; Hussaini, M., Discontinuous spectral element approximation of Maxwell’s equations, (Cockburn, B.; Kariadakis, G.; Shu, C.-W., Proceedings of the International Symposium on Discontinuous Galerkin Methods (2000), Springer) · Zbl 0957.78023
[50] Kopriva, D. A.; Woodruff, S. L.; Hussaini, M., Computation of electromagnetic scattering with a non-conforming discontinuous spectral element method, Int. J. Numer. Methods Eng., 53, 105-222 (2002) · Zbl 0994.78020
[51] Beck, A. D.; Bolemann, T.; Flad, D.; Frank, H.; Gassner, G. J.; Hindenlang, F.; Munz, C.-D., High-order discontinuous Galerkin spectral element methods for transitional and turbulent flow simulations, Int. J. Numer. Methods Fluids, 76, 8, 522-548 (2014)
[52] Fechter, S.; Munz, C.-D., A discontinuous Galerkin-based sharp-interface method to simulate three-dimensional compressible two-phase flow, Int. J. Numer. Methods Fluids, 78, 7, 413-435 (2015)
[53] Flad, D.; Frank, H.; Beck, A. D.; Munz, C.-D., A discontinuous Galerkin spectral element method for the direct numerical simulation of aeroacoustics, 20th AIAA/CEAS Aeroacoustics Conference. 20th AIAA/CEAS Aeroacoustics Conference, AIAA Pap., 2014-2740 (2014)
[54] Hindenlang, F.; Gassner, G. J.; Altmann, C.; Beck, A.; Staudenmaier, M.; Munz, C.-D., Explicit discontinuous Galerkin methods for unsteady problems, Comput. Fluids, 61, 86-93 (2012) · Zbl 1365.76117
[55] Krupp, V.; Masilamani, K.; Klimach, H.; Roller, S., Efficient coupling of fluid and acoustic interaction on massive parallel systems, (Resch, M.; Bez, W.; Focht, E.; Patel, N.; Kobayashi, H., Sustained Simulation Performance 2016 (2016), Springer International Publishing: Springer International Publishing Cham), 61-81
[56] M.H. Carpenter, C. Kennedy, Fourth-order 2N-storage Runge-Kutta schemes, NASA Report TM 109112, NASA Langley Research Center, 1994.; M.H. Carpenter, C. Kennedy, Fourth-order 2N-storage Runge-Kutta schemes, NASA Report TM 109112, NASA Langley Research Center, 1994.
[57] Bailly, C.; Bogey, C., An overview of numerical methods for acoustic wave propagation, (Wesseling, P.; Oate, E.; Périaux, J., European Conference on Computational Fluid Dynamics, ECCOMAS CFD 2006 (2006))
[58] Maday, Y.; Mavriplis, C.; Patera, A. T., Nonconforming mortar element methods: Application to spectral discretizations, (Chan, T.; Glowinski, R.; Périaux, J.; Widlund, O., Domain Decomposition Methods (1989), SIAM), 392-418 · Zbl 0692.65055
[59] Kopriva, D. A., A conservative staggered-grid Chebyshev multidomain method for compressible flows. II. A semi-structured method, J. Comput. Phys., 128, 2, 475-488 (1996) · Zbl 0866.76064
[60] Farrell, P.; Maddison, J., Conservative interpolation between volume meshes by local Galerkin projection, Comput. Methods Appl. Mech. Eng., 200, 1-4, 89-100 (2011) · Zbl 1225.76193
[61] Jiao, X.; Heath, M. T., Common-refinement-based data transfer between non-matching meshes in multiphysics simulations, Int. J. Numer. Methods Eng., 61, 14, 2402-2427 (2004) · Zbl 1075.74711
[62] Bogey, C.; Bailly, C., Computation of a high Reynolds number jet and its radiated noise using large eddy simulation based on explicit filtering, Comput. Fluids, 35, 10, 1344-1358 (2006) · Zbl 1177.76372
[63] Bogey, C.; Bailly, C.; Juvé, D., Noise investigation of a high subsonic, moderate Reynolds number jet using a compressible large eddy simulation, Theor. Comput. Fluid Dyn., 16, 4, 273-297 (2003) · Zbl 1051.76064
[64] Bogey, C.; Bailly, C., Direct computation of the sound radiated by a high-Reynolds number, subsonic round jet, (CES Workshop from CFD To CAA, 7-8 November 2002, NTUA, Athens, Greece (2002))
[65] Tam, C. K.W.; Webb, J. C., Dispersion-relation-preserving finite difference schemes for computational acoustics, J. Comput. Phys., 107, 262-281 (1993) · Zbl 0790.76057
[66] Bogey, C.; Bailly, C., Effects of inflow conditions and forcing on subsonic jet flows and noise, AIAA J., 43, 5, 1000-1007 (2005)
[67] Tam, C. K., Jet noise: Since 1952, Theor. Comput. Fluid Dyn., 10, 1-4, 393-405 (1998) · Zbl 0900.76558
[68] Billson, M.; Eriksson, L.-E.; Davidson, L., Acoustic source terms for the linearized Euler equations in conservative form, AIAA J., 43, 4, 752-759 (2005)
[69] Bae, Y.; Moon, Y. J., Effect of passive porous surface on the trailing-edge noise, Phys. Fluids, 23, 12, 126101 (2011)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.