Related papers: Improving Levenberg-Marquardt Algorithm for Neural…
We consider the problem of efficiently computing the maximum likelihood estimator in Generalized Linear Models (GLMs) when the number of observations is much larger than the number of coefficients ($n \gg p \gg 1$). In this regime,…
We propose an online learning algorithm for a class of machine learning models under a separable stochastic approximation framework. The essence of our idea lies in the observation that certain parameters in the models are easier to…
Due to the ease of modern data collection, applied statisticians often have access to a large set of covariates that they wish to relate to some observed outcome. Generalized linear models (GLMs) offer a particularly interpretable framework…
Stochastic gradient-based descent (SGD), have long been central to training large language models (LLMs). However, their effectiveness is increasingly being questioned, particularly in large-scale applications where empirical evidence…
Policy gradient algorithms are widely used in reinforcement learning and belong to the class of approximate dynamic programming methods. This paper studies two key policy gradient algorithms, the Natural Policy Gradient and the Gauss-Newton…
In this work, a new class of stochastic gradient algorithm is developed based on $q$-calculus. Unlike the existing $q$-LMS algorithm, the proposed approach fully utilizes the concept of $q$-calculus by incorporating time-varying $q$…
Policy gradient methods are a powerful family of reinforcement learning algorithms for continuous control that optimize a policy directly. However, standard first-order methods often converge slowly. Second-order methods can accelerate…
Markov logic networks (MLNs) reconcile two opposing schools in machine learning and artificial intelligence: causal networks, which account for uncertainty extremely well, and first-order logic, which allows for formal deduction. An MLN is…
Stochastic gradient descent and other first-order variants, such as Adam and AdaGrad, are commonly used in the field of deep learning due to their computational efficiency and low-storage memory requirements. However, these methods do not…
In this paper, we propose a novel optimization algorithm for training machine learning models called Input Normalized Stochastic Gradient Descent (INSGD), inspired by the Normalized Least Mean Squares (NLMS) algorithm used in adaptive…
Many attempts took place to improve the adaptive filters that can also be useful to improve backpropagation (BP). Normalized least mean squares (NLMS) is one of the most successful algorithms derived from Least mean squares (LMS). However,…
Deep learning algorithms often require solving a highly non-linear and nonconvex unconstrained optimization problem. Methods for solving optimization problems in large-scale machine learning, such as deep learning and deep reinforcement…
Gradient-based local optimization has been shown to improve results of genetic programming (GP) for symbolic regression. Several state-of-the-art GP implementations use iterative nonlinear least squares (NLS) algorithms such as the…
It has been shown numerically that the performance of the Levenberg-Marquardt algorithm can be improved by including a second order correction known as the geodesic acceleration. In this paper we give the method a more sound theoretical…
Due to the strong correlations present in quantum systems, classical machine learning algorithms like stochastic gradient descent are often insufficient for the training of neural network quantum states (NQSs). These difficulties can be…
We report an inter-comparison of some popular algorithms within the artificial neural network domain (viz., Local search algorithms, global search algorithms, higher order algorithms and the hybrid algorithms) by applying them to the…
We derive nonlinear acceleration methods based on the limited memory BFGS (L-BFGS) update formula for accelerating iterative optimization methods of alternating least squares (ALS) type applied to canonical polyadic (CP) and Tucker tensor…
Minimizing the sum of a convex function and a composite function appears in various fields. The generalized Levenberg--Marquardt (LM) method, also known as the prox-linear method, has been developed for such optimization problems. The…
Nonlinear regression methods, such as local optimization algorithms, are widely used in the extraction of nanostructure profile parameters in optical scatterometry. The success of local optimization algorithms heavily relies on the…
Recently, a Levenberg-Marquardt method with Singular Scaling matrix, called LMMSS, was proposed and successfully applied in parameter estimation in heat conduction problems, where the choice of suitable singular scaling matrix resulted in…