English

A History of Meta-gradient: Gradient Methods for Meta-learning

Machine Learning 2022-02-22 v1
Authors: Richard S. Sutton
View on arXiv ↗ PDF ↗

Abstract

The history of meta-learning methods based on gradient descent is reviewed, focusing primarily on methods that adapt step-size (learning rate) meta-parameters.

Keywords

machine learning

Cite

@article{arxiv.2202.09701,
  title  = {A History of Meta-gradient: Gradient Methods for Meta-learning},
  author = {Richard S. Sutton},
  journal= {arXiv preprint arXiv:2202.09701},
  year   = {2022}
}

Comments

3 pages of text, 54 references

Related papers

View all related →
Machine Learning · Statistics
Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace
Yoonho Lee, Seungjin Choi
2018-06-15
Machine Learning · Computer Science
Meta-learning the Learning Trends Shared Across Tasks
Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan +1
2020-10-20
Machine Learning · Computer Science
Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization
Min-Kook Suh, Seung-Woo Seo
2024-01-09
Machine Learning · Statistics
Optimizing ML Training with Metagradient Descent
Logan Engstrom, Andrew Ilyas, Benjamin Chen, Axel Feldmann +2
2025-03-19
Machine Learning · Statistics
Gradient descent revisited via an adaptive online learning rate
Mathieu Ravaut, Satya Gorti
2018-04-10
Machine Learning · Computer Science
Decoder Choice Network for Meta-Learning
Jialin Liu, Fei Chao, Longzhi Yang, Chih-Min Lin +1
2019-11-20
Machine Learning · Computer Science
Optimistic Meta-Gradients
Sebastian Flennerhag, Tom Zahavy, Brendan O'Donoghue, Hado van Hasselt +2
2023-01-10
Machine Learning · Computer Science
Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
Chelsea Finn, Sergey Levine
2018-02-16
Machine Learning · Computer Science
Differentiable Self-Adaptive Learning Rate
Bozhou Chen, Hongzhi Wang, Chenmin Ba
2022-10-20
Machine Learning · Computer Science
Gradient Descent with Provably Tuned Learning-rate Schedules
Dravyansh Sharma
2025-12-05
Machine Learning · Computer Science
MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters
Arsalan Sharifnassab, Saber Salehkaleybar, Richard Sutton
2025-07-10
Machine Learning · Computer Science
Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descent
Guangzeng Xie, Hao Jin, Dachao Lin, Zhihua Zhang
2021-04-13
Machine Learning · Statistics
A Fixed-Point of View on Gradient Methods for Big Data
Alexander Jung
2017-08-16
Machine Learning · Computer Science
Adaptive Hierarchical Hyper-gradient Descent
Renlong Jie, Junbin Gao, Andrey Vasnev, Minh-Ngoc Tran
2021-05-12
Machine Learning · Computer Science
Angle based dynamic learning rate for gradient descent
Neel Mishra, Pawan Kumar
2023-04-21
Machine Learning · Computer Science
Understanding Benign Overfitting in Gradient-Based Meta Learning
Lisha Chen, Songtao Lu, Tianyi Chen
2022-11-10
Disordered Systems and Neural Networks · Physics
Analysis of Natural Gradient Descent for Multilayer Neural Networks
Magnus Rattray, David Saad
2009-10-31
Machine Learning · Computer Science
Adaptive Gradient-Based Meta-Learning Methods
Mikhail Khodak, Maria-Florina Balcan, Ameet Talwalkar
2019-12-10
Machine Learning · Computer Science
Meta-Learning with Warped Gradient Descent
Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin +2
2020-02-19
Machine Learning · Computer Science
Learning to Accelerate by the Methods of Step-size Planning
Hengshuai Yao
2022-05-27
Machine Learning · Computer Science
Meta-Learning: A Survey
Joaquin Vanschoren
2018-10-09
Machine Learning · Computer Science
Exploring the Optimized Value of Each Hyperparameter in Various Gradient Descent Algorithms
Abel C. H. Chen
2024-09-19
Machine Learning · Computer Science
When Meta-Learning Meets Online and Continual Learning: A Survey
Jaehyeon Son, Soochan Lee, Gunhee Kim
2024-11-11
Machine Learning · Computer Science
Meta-learning approaches for few-shot learning: A survey of recent advances
Hassan Gharoun, Fereshteh Momenifar, Fang Chen, Amir H. Gandomi
2023-03-15
Machine Learning · Computer Science
Delay-adaptive step-sizes for asynchronous learning
Xuyang Wu, Sindri Magnusson, Hamid Reza Feyzmahdavian, Mikael Johansson
2022-04-12
R2 v1 2026-06-24T09:46:08.229Z