Related papers: Nonlinear Least Squares for Large-Scale Machine Le…
Non-linear least squares solvers are used across a broad range of offline and real-time model fitting problems. Most improvements of the basic Gauss-Newton algorithm tackle convergence guarantees or leverage the sparsity of the underlying…
We describe, implement and test a novel method for training neural networks to estimate the Jacobian matrix $J$ of an unknown multivariate function $F$. The training set is constructed from finitely many pairs $(x,F(x))$ and it contains no…
An effective numerical method is presented for optimizing model parameters that can be applied to any type of system of non-linear equations and any number of data-points, which does not require explicit formulation of the objective…
We investigate the nonlinear regression problem under L2 loss (square loss) functions. Traditional nonlinear regression models often result in non-convex optimization problems with respect to the parameter set. We show that a convex…
We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer…
Nonlinear least-squares problems are a special class of unconstrained optimization problems in which their gradient and Hessian have special structures. In this paper, we exploit these structures and proposed a matrix-free algorithm with a…
Accurately estimating parameters in complex nonlinear systems is crucial across scientific and engineering fields. We present a novel approach for parameter estimation using a neural network with the Huber loss function. This method taps…
This paper studies sparse nonlinear least squares problems, where the Jacobian matrices are unavailable or expensive to compute, yet have some underlying sparse structures. We construct the Jacobian models by the $ \ell_1 $ minimization…
This work considers the problem of learning the Markov parameters of a linear system from observed data. Recent non-asymptotic system identification results have characterized the sample complexity of this problem in the single and…
This paper proposes several novel optimization algorithms for minimizing a nonlinear objective function. The algorithms are enlightened by the optimal state trajectory of an optimal control problem closely related to the minimized objective…
Learning expressive probabilistic models correctly describing the data is a ubiquitous problem in machine learning. A popular approach for solving it is mapping the observations into a representation space with a simple joint distribution,…
This is a brief tutorial on the least square estimation technique that is straightforward yet effective for parameter estimation. The tutorial is focused on the linear LSEs instead of nonlinear versions, since most nonlinear LSEs can be…
The problem of prediction in functional linear regression is conventionally addressed by reducing dimension via the standard principal component basis. In this paper we show that an alternative basis chosen through weighted least-squares,…
We study stochastic optimization of nonconvex loss functions, which are typical objectives for training neural networks. We propose stochastic approximation algorithms which optimize a series of regularized, nonlinearized losses on large…
This work provides simple algorithms for multi-class (and multi-label) prediction in settings where both the number of examples n and the data dimension d are relatively large. These robust and parameter free algorithms are essentially…
Nonlinear dynamic models are widely used for characterizing functional forms of processes that govern complex biological pathway systems. Over the past decade, validation and further development of these models became possible due to data…
Linear Least Squares is a very well known technique for parameter estimation, which is used even when sub-optimal, because of its very low computational requirements and the fact that exact knowledge of the noise statistics is not required.…
Nonlinear regression analysis is a popular and important tool for scientists and engineers. In this article, we introduce theories and methods of nonlinear regression and its statistical inferences using the frequentist and Bayesian…
The problem of fitting experimental data to a given model function $f(t; p_1,p_2,\dots,p_N)$ is conventionally solved numerically by methods such as that of Levenberg-Marquardt, which are based on approximating the Chi-squared measure of…
Least squares form one of the most prominent classes of optimization problems, with numerous applications in scientific computing and data fitting. When such formulations aim at modeling complex systems, the optimization process must…