Related papers: Convergence of Stochastic Gradient Methods for Wid…

A physics informed neural network (PINN) incorporates the physics of a system by satisfying its boundary value problem through a neural network's loss function. The PINN approach has shown great success in approximating the map between the…

Physics Informed Neural Networks (PINNs) are shown to be a promising method for the approximation of Partial Differential Equations (PDEs). PINNs approximate the PDE solution by minimizing physics-based loss functions over a given domain.…

Backpropagation with gradient descent is a common optimization strategy employed by most neural network architectures in machine learning. However, finding optimal hyperparameters to guide training has proven challenging. While it is widely…

Deep learning-based numerical schemes such as Physically Informed Neural Networks (PINNs) have recently emerged as an alternative to classical numerical schemes for solving Partial Differential Equations (PDEs). They are very appealing at…

This work considers stochastic Galerkin approximations of linear elliptic partial differential equations (PDEs) with stochastic forcing terms and stochastic diffusion coefficients, that cannot be bounded uniformly away from zero and…

Physics-informed neural network (PINN) is a powerful emerging method for studying forward-inverse problems of partial differential equations (PDEs), even from limited sample data. Variable coefficient PDEs, which model real-world phenomena,…

Bilevel optimization problems are receiving increasing attention in machine learning as they provide a natural framework for hyperparameter optimization and meta-learning. A key step to tackle these problems is the efficient computation of…

Physics-informed neural networks (PINNs) have emerged as a promising deep learning method, capable of solving forward and inverse problems governed by differential equations. Despite their recent advance, it is widely acknowledged that…

Learning rates in stochastic neural network training are currently determined a priori to training, using expensive manual or automated iterative tuning. This study proposes gradient-only line searches to resolve the learning rate for…

Physics-informed deep learning has drawn tremendous interest in recent years to solve computational physics problems, whose basic concept is to embed physical laws to constrain/inform neural networks, with the need of less data for training…

Physics-Informed Neural Networks (PINNs) frequently encounter difficulties in accurately resolving shock waves within high-speed compressible flows, a failure largely attributed to the "gradient pathology" arising from extreme stiffness at…

We investigate the training of Physics-Informed Neural Networks (PINNs) from a control-theoretic perspective. Using gradient descent with resampling, we interpret the training dynamics as asymptotically equivalent to a stochastic…

Physics-Informed Neural Networks (PINNs) have demonstrated considerable success in solving complex fluid dynamics problems. However, their performance often deteriorates in regimes characterized by steep gradients, intricate boundary…

Existing analyses of optimization in deep learning are either continuous, focusing on (variants of) gradient flow, or discrete, directly treating (variants of) gradient descent. Gradient flow is amenable to theoretical analysis, but is…

We present two stochastic descent algorithms that apply to unconstrained optimization and are particularly efficient when the objective function is slow to evaluate and gradients are not easily obtained, as in some PDE-constrained…

Neural network optimization remains one of the most consequential yet poorly understood challenges in modern AI research, where improvements in training algorithms can lead to enhanced feature learning in foundation models,…

Physics-informed neural networks (PINNs) have emerged as a promising numerical method based on deep learning for modeling boundary value problems, showcasing promising results in various fields. In this work, we use PINNs to discretize…

Physics-informed neural networks (PINNs) have emerged as a powerful approach for solving partial differential equations (PDEs) by training neural networks with loss functions that incorporate physical constraints. In this work, we introduce…

Recently developed physics-informed neural network (PINN) has achieved success in many science and engineering disciplines by encoding physics laws into the loss functions of the neural network, such that the network not only conforms to…

In this paper, we consider a general stochastic optimization problem which is often at the core of supervised learning, such as deep learning and linear classification. We consider a standard stochastic gradient descent (SGD) method with a…