Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound
Machine Learning
2020-02-19 v2 Artificial Intelligence
Machine Learning
Abstract
We improve the over-parametrization size over two beautiful results [Li and Liang' 2018] and [Du, Zhai, Poczos and Singh' 2019] in deep learning theory.
Cite
@article{arxiv.1906.03593,
title = {Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound},
author = {Zhao Song and Xin Yang},
journal= {arXiv preprint arXiv:1906.03593},
year = {2020}
}
Related papers
View all related →
Machine Learning · Computer Science
On the Power of Over-parametrization in Neural Networks with Quadratic Activation
Simon S. Du, Jason D. Lee
2018-06-18
Machine Learning · Computer Science
Nearly Minimal Over-Parametrization of Shallow Neural Networks
Armin Eftekhari, ChaeHwan Song, Volkan Cevher
2019-10-30
Machine Learning · Statistics
Empirical Study of the Benefits of Overparameterization in Learning Latent Variable Models
Rares-Darius Buhai, Yoni Halpern, Yoon Kim, Andrej Risteski +1
2020-07-17
Machine Learning · Computer Science
Online Learning for the Random Feature Model in the Student-Teacher Framework
Roman Worschech, Bernd Rosenow
2023-04-10
Machine Learning · Computer Science
Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Time
Zhao Song, Lichen Zhang, Ruizhe Zhang
2023-11-27
Machine Learning · Computer Science
Subquadratic Overparameterization for Shallow Neural Networks
Chaehwan Song, Ali Ramezani-Kebrya, Thomas Pethick, Armin Eftekhari +1
2021-11-04
Machine Learning · Computer Science
Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks
Xiangyu Chang, Yingcong Li, Samet Oymak, Christos Thrampoulidis
2020-12-17
Machine Learning · Computer Science
Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamics
Soo Min Kwon, Zekai Zhang, Dogyoon Song, Laura Balzano +1
2024-03-13
Machine Learning · Computer Science
How Does Overparameterization Affect Features?
Ahmet Cagri Duzgun, Samy Jelassi, Yuanzhi Li
2024-07-02
Computer Vision and Pattern Recognition · Computer Science
Mapping Networks
Lord Sen, Shyamapada Mukherjee
2026-02-24
Machine Learning · Computer Science
Mathematical Models of Overparameterized Neural Networks
Cong Fang, Hanze Dong, Tong Zhang
2020-12-29
Machine Learning · Computer Science
How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
Zixiang Chen, Yuan Cao, Difan Zou, Quanquan Gu
2022-01-03
Machine Learning · Computer Science
Convex Formulation of Overparameterized Deep Neural Networks
Cong Fang, Yihong Gu, Weizhong Zhang, Tong Zhang
2019-11-19
Number Theory · Mathematics
Hausdorff dimensions of very well intrinsically approximable subsets of quadratic hypersurfaces
Lior Fishman, Keith Merrill, David Simmons
2017-09-18
Machine Learning · Computer Science
Optimal Approximations Made Easy
Mónika Csikós, Nabil H. Mustafa
2022-09-02
Machine Learning · Computer Science
Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations
Cong Fang, Hanze Dong, Tong Zhang
2019-10-28
Data Structures and Algorithms · Computer Science
Randomized and Deterministic Attention Sparsification Algorithms for Over-parameterized Feature Dimension
Yichuan Deng, Sridhar Mahadevan, Zhao Song
2023-04-11
Computer Vision and Pattern Recognition · Computer Science
Boosting Pruned Networks with Linear Over-parameterization
Yu Qian, Jian Cao, Xiaoshuang Li, Jie Zhang +2
2024-01-01
Machine Learning · Computer Science
Learning Compact Neural Networks with Deep Overparameterised Multitask Learning
Shen Ren, Haosen Shi
2023-08-28
Machine Learning · Computer Science
Understanding How Over-Parametrization Leads to Acceleration: A case of learning a single teacher neuron
Jun-Kun Wang, Jacob Abernethy
2021-09-29
Machine Learning · Computer Science
Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma +1
2021-03-12
Machine Learning · Computer Science
Theoretical insights into the optimization landscape of over-parameterized shallow neural networks
Mahdi Soltanolkotabi, Adel Javanmard, Jason D. Lee
2022-08-25
Data Structures and Algorithms · Computer Science
More Asymmetry Yields Faster Matrix Multiplication
Josh Alman, Ran Duan, Virginia Vassilevska Williams, Yinzhan Xu +2
2024-10-22
Machine Learning · Computer Science
On Optimal Early Stopping: Over-informative versus Under-informative Parametrization
Ruoqi Shen, Liyao Gao, Yi-An Ma
2022-02-25
Machine Learning · Computer Science
The Low-Rank Simplicity Bias in Deep Networks
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung +2
2023-03-24