Past Event: Babuška Forum

Rethinking Physics Informed Neural Networks

Amir Gholami, Research scientist, UC Berkeley

Abstract

One of the next frontiers in machine learning is to find a way to incorporate physical invariances into learning. Invariances can help in various ways including regularizing the model in the presence of limited training data, and/or help improve convergence and generalization. Building these invariances into the model/training could be quite impactful for a range of real-world deployments (e.g. real-world deployment of an agent), as well as ML problems in science/engineering. However, there are several open problems and challenges in incorporating physical constraints into learning. One popular approach is to incorporate physical domain knowledge as soft constraints on an empirical loss function and use existing machine learning methodologies to train the model, commonly known as Physics Informed Neural Networks (PINNs). However, this soft regularization, which involves differential operators, can introduce a number of subtle problems, including making the problem ill-conditioned. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena even for simple PDEs. In particular, we analyze several distinct situations of widespread physical interest, including learning differential equations with convection, reaction, and diffusion operators. Importantly, we show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN’s setup makes the loss landscape very hard to optimize. We then describe two promising solutions to address these failure modes. The first approach is to use curriculum regularization, where the PINN's loss term starts from a simple PDE regularization and becomes progressively more complex as the NN gets trained. The second approach is to pose the problem as a sequence-to-sequence learning task, rather than learning to predict the entire space-time at once. Extensive testing shows that we can achieve up to 1-2 orders of magnitude lower error with these methods as compared to regular PINN training.

Related Papers:

• Characterizing possible failure modes in physics-informed neural networks (NeurIPS'21; accepted);
https://arxiv.org/pdf/2109.01050.pdf

• AdaHessian: An Adaptive Second Order Optimizer for Machine Learning (AAAI'21);
https://arxiv.org/pdf/2006.00719.pdf

• ANODEV2: A Coupled Neural ODE Evolution Framework (NeurIPS'19);
https://arxiv.org/pdf/1906.04596.pdf

Biography

Amir Gholami is a research scientist in RiseLab and BAIR at UC Berkeley. He received his PhD from UT Austin, working on large scale 3D image segmentation, a research topic which received UT Austin’s best doctoral dissertation award in 2018. He is a Melosh Medal finalist, the recipient of Amazon Machine Learning Research Award in 2020, best student paper award in SC'17, Gold Medal in the ACM Student Research Competition, best student paper finalist in SC’14. Amir was part of the Nvidia team that for the first time made low precision neural network training possible (FP16), enabling more than 10x increase in compute power through tensor cores. That technology has been widely adopted in GPUs today. Amir's current research focuses on efficient AI, AutoML, and scalable training of Neural Network models.