Learning Function Operators with Neural Networks

Samuel Burbulla, Senior AI Researcher at appliedAI Institute, will give an overview of recent advances in operator learning and their application to numerical simulation.

Abstract

Neural networks can learn function operators: non-linear mappings of continuous functions. This is particularly interesting in scientific machine learning (SciML), where physical quantities are often represented as functions in space or time. Utilizing neural networks for function operator learning can drastically accelerate numerical simulations, offering speedups over traditional numerical solvers by orders of magnitude. This overview talk addresses the latest advancements in operator learning, focusing on architectures such as DeepONets, Fourier Neural Operators (FNOs), and other recent developments in the field. Along with hands-on examples, we will outline how these methods enable discretization-invariant representations, super-resolution capabilities, and physics-informed training.

References

[Kov23N]

Neural Operator: Learning Maps Between Function Spaces With Applications to PDEs, Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar.

2023

The classical development of neural networks has primarily focused on learning mappings between finite dimensional Euclidean spaces or finite sets. We propose a generalization of neural networks to learn operators, termed neural operators, that map between infinite dimensional function spaces. We formulate the neural operator as a composition of linear integral operators and nonlinear activation …

[Wan21L]

Learning the solution operator of parametric partial differential equations with physics-informed DeepONets, Sifan Wang, Hanwen Wang, Paris Perdikaris.

Sep 2021

Partial differential equations (PDEs) play a central role in the mathematical analysis and modeling of complex dynamic processes across all corners of science and engineering. Their solution often requires laborious analytical or computational tools, associated with a cost that is markedly amplified when different scenarios need to be investigated, for example, corresponding to different initial …

[Lu21L]

Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, George Em Karniadakis.

Mar 2021

It is widely known that neural networks (NNs) are universal approximators of continuous functions. However, a less known but powerful result is that a NN with a single hidden layer can accurately approximate any nonlinear continuous operator. This universal approximation theorem of operators is suggestive of the structure and potential of deep neural networks (DNNs) in learning continuous …

[Rai19P]

Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, M. Raissi, P. Perdikaris, G. E. Karniadakis.

Feb 2019

We introduce physics-informed neural networks – neural networks that are trained to solve supervised learning tasks while respecting any given laws of physics described by general nonlinear partial differential equations. In this work, we present our developments in the context of solving two main classes of problems: data-driven solution and data-driven discovery of partial differential …

[Bar23R]

Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning, Francesca Bartolucci, Emmanuel Bezenac, Bogdan Raonic, Roberto Molinaro, Siddhartha Mishra, Rima Alaifari.

Nov 2023

Recently, operator learning, or learning mappings between infinite-dimensional function spaces, has garnered significant attention, notably in relation to learning partial differential equations from data. Conceptually clear when outlined on paper, neural operators necessitate discretization in the transition to computer implementations. This step can compromise their integrity, often causing them …

[Che95U]

Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems, Tianping Chen, Hong Chen.

Jul 1995

The purpose of this paper is to investigate neural network capability systematically. The main results are: 1) every Tauber-Wiener function is qualified as an activation function in the hidden layer of a three-layered neural network; 2) for a continuous function to be a Tauber-Wiener function, the necessary and sufficient condition is that it is not a polynomial; 3) the capability of approximating …

[Kar21P]

Physics-informed machine learning, George Em Karniadakis, Ioannis G. Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, Liu Yang.

May 2021

Despite great progress in simulating multiphysics problems using the numerical discretization of partial differential equations (PDEs), one still cannot seamlessly incorporate noisy data into existing algorithms, mesh generation remains complex, and high-dimensional problems governed by parameterized PDEs cannot be tackled. Moreover, solving inverse problems with hidden physics is often …

[Lam23G]

Learning skillful medium-range global weather forecasting., Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mohamed, Peter Battaglia.

Nov 2023

Global medium-range weather forecasting is critical to decision-making across many social and economic domains. Traditional numerical weather prediction uses increased compute resources to improve forecast accuracy, but cannot directly use historical weather data to improve the underlying model. We introduce a machine learning-based method called "GraphCast", which can be trained directly from …

[Li20F]

Fourier Neural Operator for Parametric Partial Differential Equations, Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar.

Oct 2020

The classical development of neural networks has primarily focused on learning mappings between finite-dimensional Euclidean spaces. Recently, this has been generalized to neural operators that learn mappings between function spaces. For partial differential equations (PDEs), neural operators directly learn the mapping from any functional parametric dependence to the solution. Thus, they learn an …

[Wei23S]

Super-Resolution Neural Operator, Min Wei, Xuesong Zhang.

Mar 2023

We propose Super-resolution Neural Operator (SRNO), a deep operator learning framework that can resolve high-resolution (HR) images at arbitrary scales from the low-resolution (LR) counterparts. Treating the LR-HR image pairs as continuous functions approximated with different grid sizes, SRNO learns the mapping between the corresponding function spaces. From the perspective of approximation …

[Zha23B]

BelNet: basis enhanced learning, a mesh-free neural operator, Zecheng Zhang, Leung Wing Tat, Hayden Schaeffer.

Aug 2023

Operator learning trains a neural network to map functions to functions. An ideal operator learning framework should be mesh-free in the sense that the training does not require a particular choice of discretization for the input functions, allows for the input and output functions to be on different domains, and is able to have different grids between samples. We propose a mesh-free neural …

[Lu22C]

A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data, Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, George Em Karniadakis.

Apr 2022

Neural operators can learn nonlinear mappings between function spaces and offer a new simulation paradigm for real-time prediction of complex dynamics for realistic diverse applications as well as for system identification in science and engineering. Herein, we investigate the performance of two neural operators, which have shown promising results so far, and we develop new practical extensions …

Abstract

References

In this series →