An exploratory study on machine learning to couple numerical solutions of partial differential equations. (English) Zbl 1462.65217

The authors propose a machine learning based algorithm for the solution of dynamical systems with governing equations that can be written as PDEs/coupled PDEs to understand the most complex coupled multi-physics and multiscale phenomena. The coupled governing equations involving the multiscale and multi-physical characteristics are solved with high efficiency and high numerical stability. If we could use machine learning to speed up the process of solving such complex phenomena, it could herald a new era in doing a whole lot of good for scientific inquiry and engineering. One of the key features of the machine learning approach is the fact that, unlike other commonly used numerical approaches such as FDM/FEM methods, the calculation efficiency is superlative. As such, it does not suffer (as much as other numerical methods) from the curse of dimensionality associated with high-dimensional PDEs and PDE systems. The paper represents a paradigm shift from the traditional approach based on grids to a machine-learning approach that couples numerical solutions of PDEs. It solves PDEs using domain decomposition as in a conventional method but develops and trains artificial neural networks (ANN) to couple the PDEs at their interfaces, leading to their solutions in the whole domains. The key concepts and algorithms for the ML coupling are discussed via computation of coupled Poisson equations and coupled advection-diffusion equations as model equations. One can look at the results provided in this work as a preliminary finding, and substantial research is required for extending these frameworks to complex multi-dimensional, multiscale problems. The authors, based on their research, are of the view that this exploratory study indicates that the ML paradigm is promising in terms of feasibility and performance and deserves further research. The paper surveys ongoing research on the coupling of partial differential equations and pinpoints that progress in theoretical and applied research is limited because of the complexity in coupling PDEs, especially different types of PDEs. One of the key findings of this survey is: further progress in coupling PDEs, especially distinct PDEs, via existing approaches is hindered due to the difficulties stemming from the fact that the coupling is a heterogeneous domain decomposition problem; it involves different PDEs, numerical methods, and computational meshes. The difficulties, the authors argue, are hard to overcome in a conventional coupling approach, which directly exchanges PDEs’ solutions at their interfaces according to certain interface conditions. Taking inspiration from the fact that rapid progress of machine learning (ML) is taking place also in PDE-based learning of fluid flows, the authors explore the idea of coupling PDEs via machine learning forecasting that coupling PDEs via ML methods may help overcome these difficulties and lead to a new avenue to accurate simulation of real-world multiscale and multi-physics problems. The first part of the technical section begins with the boundary value problem of the Poisson equation, over a domain, which is divided into two overlapping subdomains with interfaces. An ML Schwarz iteration algorithm to solve the problem is presented. As in a standard approach, the ML solution over the interface zone is constructed as a function of neural network function, and \(\alpha\) denotes its parameters, including weights and biases. The first thing to understand here is that neural networks are fundamentally function approximators. Conveniently, this function approximation process is what we need to solve a PDE. Since an ML solution, \(s\), is a neural network function, a key to its construction becomes to find the best weights and biases, within the network. In correspondence, this leads to a minimization problem. Thankfully, researchers who have made a breakthrough and developed software packages that can efficiently solve the minimization problem. The ML Schwarz iterative algorithm is executed using MatLab, and the resulting ML solution, being the neural network solution in subdomains. It is seen that the solution obtained with the ML method matches the exact solution, and the error mainly occurs near the interfaces. Interestingly, it is noticed that the ML Schwarz iteration may converge faster than the classic Schwarz iteration. In order to illustrate the feasibility and performance of the ML-based coupling methods proposed above, a numerical example is considered. In the second part, the paper considers an initial value problem of two coupled advection-diffusion equations over a domain, which is divided into two sub-domains that overlap with each other with interfaces. Three training approaches in terms of the place where the initial condition resides are explored. Two numerical experiments are performed to assess the feasibility and performance, first considering two identical one dimension equations and second a heterogeneous problem. Based on the findings of the experiments, it is observed that ML interface solutions are not merely repeating or data fitting. They could have certain capability of predictions; with input different from those used to train them, the interface solutions can predict output with accuracy. To illustrate and explore such predictability, experiments are presented for the above sections’ problems, but with perturbed or different initial and boundary conditions. Highlights of the paper: In ML coupling frameworks of model equations, numerical solutions in subdomains are coupled via an ML solution in between. This coupling is different from conventional coupling methods, in which the numerical solutions are coupled directly with each other. It is expected that such a difference could enable the ML-based methods to exhibit advantages over the conventional methods in the following ways.
An ML approach may avoid difficulties/singularities encountered in a conventional approach, e.g., non-physical solutions at interfaces.
Without Schwarz iteration, ML coupling works for the advection-diffusion equations, or the Schwarz iteration is unnecessary for computation.
The approach of coupling PDEs via ML is built on ML methods’ capabilities, which are anticipated to bear a solid foundation in mathematics. Empirical results are documented using tables and graphs and the paper places strong emphasis on both algorithmic and mathematical aspects.
Finally and most importantly, it is worth remarking that all neural networks are rarely a “one-size-fits-all” tool. Just as is the case with numerical methods, they need to be modified based on the problem. Continual experimentation and reflection is the key to improving results, but a solid understanding of the underlying processes is vital to avoiding pitfalls in solving numerically.
Observations and comments 1) Machine learning seems to be a powerful tool for high-dimensional problems. The coupling of model equations has infinitely many variables rendering them high dimensional. 2) In spite of vastly expanded limits on computational power, efforts are on to understand the most complex coupled multiphysics and multiscale phenomena using newer and innovative techniques. Discovery of ML based methods is a step in the right direction. 3) The iterative training process of neural networks solves an optimization problem that finds for parameters (weights and biases) that result in a minimum error or loss when evaluating the numerical examples. The error of the ML solution primarily comes from the error of the training data against the exact solution. 4) The accuracy of the solution is governed by how the information is exchanged between these solvers at the interface, and several methods have been devised for such coupling problems. it is seen that the ML solution is close to the numerical solution, and larger error occurs around the interfaces. 5) When talking about minimization in the context of neural networks, we are discussing non-convex optimization. Thanks to those researchers who made a breakthrough and developed software packages that can efficiently solve the minimization problem. 6) Although numerical methods have been widely used with good performance, researchers are still searching for new methods for solving partial differential equations. In recent years, machine learning has achieved great success in many diverse fields, such as image classification and natural language processing. Studies have shown that artificial neural networks have powerful function-fitting (approximating) capabilities and have great potential in the study of partial differential equations. Though exploratory in nature, the paper aims to show that ML paradigm is promising in terms of feasibility and performance and deserves further research. 7) ML based methods can also be interpreted in a broad sense as domain decomposition methods, where two solvers simultaneously advance the multiscale and multi-physics problems, and the information is exchanged across the interface between solvers. 8) This study is exploratory and preliminary, and further research is necessary on the ML coupling to explore the above listed potential advantages via systematic numerical and theoretical analyses. Nevertheless, this paper indicates that the ML coupling is promising, and, hopefully, it will attract the community’s attention to the topic and its further study. 9) Deep learning has achieved remarkable success in diverse applications; however, its use in solving partial differential equations (PDEs) has emerged only recently. Therefore, the present study being exploratory and preliminary, further research is necessary on the ML coupling to explore the above listed potential advantages via systematic numerical and theoretical analyses. Nevertheless, by the authors’ own admission, this paper indicates that the ML coupling is promising, and, hopefully, it will attract the community’s attention to the topic and its further study. 10) A major challenge in multiscale as well as multi-physics modelling lies in coupling the sub-models such that the overall model is both accurate enough to be scientifically relevant and reproducible, and efficient enough to be executed conveniently by modern computing resources. Precisely for this reason, research has been initiated on the mathematical foundation and mechanism of ML.


65N55 Multigrid methods; domain decomposition for boundary value problems involving PDEs
65N06 Finite difference methods for boundary value problems involving PDEs
68T07 Artificial neural networks and deep learning
35J05 Laplace operator, Helmholtz equation (reduced wave equation), Poisson equation
35K20 Initial-boundary value problems for second-order parabolic equations
Full Text: DOI arXiv


[1] Schwarz, H. A., Über einige abbildungsaufgaben, Ges Abh, 11, 65-83 (1869)
[2] Califano, G.; Conte, D., Optimal Schwarz waveform relaxation for fractional diffusion-wave equations, Appl Numer Math, 127, 125141 (2018)
[3] Gastaldi, F.; Guarteroni, A., On the coupling of hyperbolic and parabolic systems: analytical and numerical approach, Appl Numer Math, 6, 3-31 (1989/90)
[4] Gander, M. J.; Stuart, A. M., Space-time continuous analysis of waveform relaxation for the heat equation, SIAM J Sci Comput, 19, 2014-2031 (1998) · Zbl 0911.65082
[5] Martin, V., An optimized Schwarz waveform relaxation method for the unsteady convection diffusion equation in two dimensions, Computers & Fluids, 33, 829837 (2004)
[6] Gander, M. J.; Halpern, L., Optimized Schwarz waveform relaxation methods for advection diffusion problems, SIAM J Numer Anal, 45, 666-697 (2007) · Zbl 1140.65063
[7] Berger, M. J., On conservation at grid interfaces, SIAM J Numer Anal, 24, 967-984 (1987) · Zbl 0633.65086
[8] Tang, H. S.; Zhou, T., On non-conservative algorithms for grid interfaces, SIAM J Numer Anal, 37, 173-193 (1999) · Zbl 0948.65087
[9] Keyes, D. E., Multiphysics simulations: challenges and opportunities, Int J High Performance Computing Application, 27, 1, 4-86 (2012)
[10] Tang, H. S.; Haynes, R.; Houzeaux, G., A review of domain decomposition methods for simulation of fluid flows: concepts, algorithms, and applications, Arch Computat Methods Eng (2020)
[11] Minami, S.; Kawai, H.; Yoshimura, S., A monolithic approach based on the balancing domain decomposition method for acoustic fluid-structure interaction, J Appl Mech, 79, 1, 010906 (2012)
[12] Tang, H. S.; Qu, K.; Wu, X. G., An overset grid method for integration of fully 3D fluid dynamics and geophysical fluid dynamics models to simulate multiphysics coastal ocean flows, J Comput Phys, 273, 548-571 (2014) · Zbl 1351.86007
[13] Blayo, E., About interface conditions for coupling hydrostatic and nonhydrostatic Navier-stokes flows, Discrete and Continuous Dynamical Systems Series, S, 9(5), 1565-1574 (2016) · Zbl 1388.76048
[14] Qu, K.; Tang, H. S.; Agrawal, A., Integration of fully 3D fluid dynamics and geophysical fluid dynamics models for multiphysics coastal ocean flows: simulation of local complex free-surface phenomena, Ocean Modell, 135, 14-30 (2019)
[15] Eisenmann, M.; Hansen, E., Convergence analysis of domain decomposition based time integrators for degenerate parabolic equations, Numerische Mathematik, 140, 913938 (2018)
[16] Pärt-Enander, E.; Sjögreen, B., Conservative and nonconservative interpolation between overlapping grids for finite volume solutions of hyperbolic problems, Comput & Fluids, 23, 551-574 (1994) · Zbl 0813.76071
[17] Wu, Z. L., On uniqueness of steady state solutions for difference equations on overlapping grids, SIAM J of Numer Anal, 33(4), 1336-1357 (1996) · Zbl 0859.65090
[18] Tang, H. S.; Zhang, D. L.; Lee, C. H., Comments on algorithms for grid interfaces in simulating Euler flows, Comm Nonlinear Science and Numerical Simulation, 1, 50-54 (1996)
[19] Tang, H. S.; Qu, K.; Wu, X. G.; Zhang, Z. K., Domain decomposition for a hybrid fully 3D fluid dynamics and geophysical fluid dynamics modeling system: a numerical experiment on a transient sill flow (2016), Domain Decomposition Methods in Science and Engineering XXII, 407-414. Lecture Notes in Computational Science and Engineering, XXII. Ed: Dickopf, T and Gander, MJ and Halpern, L and Krause, R and Pavarino, LF. Springer · Zbl 1384.86006
[20] In: haynes R. et al. (eds)
[21] Tang, H. S.; Jones, C.; Sotiropoulos, F., An overset grid method for 3d unsteady incompressible flows, J Comput Phys, 191, 567-600 (2003) · Zbl 1134.76435
[22] Chen, R. L.; Wu, Y. Q.; Yan, Z.; Zhao, Y.; Cai, X.-C., A parallel domain decomposition method for 3D unsteady incompressible flows at high Reynolds number, J Sci Comput, 58, 275-289 (2014) · Zbl 1298.76057
[23] Yassin, N. I.R.; Omran, S.; El Houby, E. M.F.; M., E.; Allam, H., Machine learning techniques for breast cancer computer aided diagnosis using different image modalities: a systematic review, Comput Methods Programs Biomed, 156, 25-45 (2018)
[24] Park, W.; Kim, S.; Kim, K. L.; Kim, J., Alphago’s decision making, J Logics and Their Applications, 6, 105-155 (2019)
[25] Tompson, J.; Schlachter, K.; Sprechmann, P.; Perlin, K., Accelerating eulerian fluid simulationwith convolutional networks, PMLR, 70, 3424-3433 (2017)
[26] Raissi, M.; Wang, Z.; Triantafyllou, M. S.; Karniadakis, G. E., Deep learning of vortex-induced vibrations, J Fluid Mech, 861, 119-137 (2019) · Zbl 1415.76177
[27] Lee, S.; You, D., Prediction of laminar vortex shedding over a cylinder using deep learning. arxiv:1712.07854 [physics.flu-dyn], J Fluid Mechanics, submitted (2017)
[28] Wiewel, S.; Becher, M.; Thuerey, N., Latent space physics: towards learning the temporal evolution of fluid flow, Eurographics, 38 (2019)
[29] Sirignan, J.; Spiliopoulos, K., DGM: A deep learning algorithm for solving partial differential equations, J Comput Phys, 375, 1339-1364 (2018) · Zbl 1416.65394
[30] MathWorks. Matlab. https://www.mathworks.com/products/matlab.html.
[31] PyTorch. From research to production. https://pytorch.org/
[32] Tensorflow. An end-to-end open source machine learning platform. https://www.tensorflow.org/.
[33] Pawar S., Ahmed S.E., San M.. Interface learning in fluid dynamics: statistical inference of closures within micro-macro coupling models. arxiv:2008.04490v1 [physics.comp-ph] 11 aug 2020. 2020.
[34] Tang H.S., Liu Y.J., Grossberg M. Research proposals. 1) collaborative research: CDS& E: A modeling system and simulation for emerging complex flow problems in coastal waters. NSF, Sept., 2019; 2) A machine-learning framework to couple numerical solutions of partial differential equations towards flow simulations, DOE, Apr. 2020.
[35] Toselli, A.; Widlund, O., Domain Decomposition Methods Algorithms and Theory (2005), Springer · Zbl 1069.65138
[36] Lagaris, I. E.; Likas, A.; Fotiadis, D. I., Artificial neural networks for solving ordinary and partial differential equations, IEEE Trans Neural Networks, 9, 987-1000 (1998)
[37] Baymani, M.; Effati, S.; Niazmand, H.; Kerayechian, A., Artificial neural network method for solving the NavierStokes equations, Neural Comput & Applic, 26, 765773 (2015)
[38] Kou, J.; Zhang, W., Multi-fidelity modeling framework for nonlinear unsteady aerodynamics of airfoils, Appl Math Model, 76, 832855 (2019)
[39] Liu, D.; Wang, Y., Multi-fidelity physics-constrained neural network and its application in materials modeling, J Mech Design, 141(12), 121403 (2019)
[40] Mojtabi, A.; Deville, M., One-dimensional linear advectiondiffusion equation:analytical and finite element solutions, Computers & Fluids, 107, 189-195 (2015)
[41] NSF. NSF-Simons research collaborations on the mathematical and scientific foundations of deep learning (MoDL), Division of Mathematical Sciences, solicitation 20-540. 2020.
[42] DOE, Advanced Scientific Computing Research. Scientific machine learning for modeling and simulations: DE-FOA-2319. 2020.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.