EOC 处的 MLPs：NTK 的集中现象
机器学习
2025-01-27 v1 机器学习
摘要
我们研究在参数 θ ∈ Θ \theta \in \Theta θ ∈ Θ l l l N : R m 0 × Θ → R m l N : \mathbb{R}^{m_0} \times \Theta \to \mathbb{R}^{m_l} N : R m 0 × Θ → R m l K θ : R m 0 × R m 0 → R m l × m l K_\theta : \mathbb{R}^{m_0} \times \mathbb{R}^{m_0} \to \mathbb{R}^{m_l \times m_l} K θ : R m 0 × R m 0 → R m l × m l ϕ ( s ) = a s + b ∣ s ∣ \phi(s) = a s + b \vert s \vert ϕ ( s ) = a s + b ∣ s ∣ a , b ∈ R a,b \in \mathbb{R} a , b ∈ R K θ ( x i 1 , x i 2 ) K_\theta(x_{i_1},x_{i_2}) K θ ( x i 1 , x i 2 ) { x 1 , ⋯ , x n } ⊂ R m 0 \{x_1,\cdots,x_n\} \subset \mathbb{R}^{m_0} { x 1 , ⋯ , x n } ⊂ R m 0 i 1 , i 2 ∈ [ 1 : n ] i_1,i_2 \in [1:n] i 1 , i 2 ∈ [ 1 : n ] K ( θ ) = [ 1 n K θ ( x i 1 , x i 2 ) : i 1 , i 2 ∈ [ 1 : n ] ] ∈ R n m l × n m l K(\theta) = [\frac{1}{n} K_\theta(x_{i_1},x_{i_2}) : i_1,i_2 \in [1:n]] \in \mathbb{R}^{nm_l \times nm_l} K ( θ ) = [ n 1 K θ ( x i 1 , x i 2 ) : i 1 , i 2 ∈ [ 1 : n ]] ∈ R n m l × n m l K ∞ ∈ R n m l × n m l \overset{\scriptscriptstyle\infty}{K} \in \mathbb{R}^{nm_l \times nm_l} K ∞ ∈ R n m l × n m l m k = k 2 m m_k = k^2 m m k = k 2 m m ∈ N + 1 m \in \mathbb{N}+1 m ∈ N + 1 P ( ∥ K ( θ ) − K ∞ ∥ ≤ O ( ( Δ ϕ − 2 + m l 1 2 l ) κ ϕ 2 m − 1 2 ) ) ≥ 1 − O ( m − 1 ) \mathbb{P}( \| K(\theta) - \overset{\scriptscriptstyle\infty}{K} \| \leq O((\Delta_\phi^{-2} + m_l^{\frac{1}{2}} l) \kappa_\phi^2 m^{-\frac{1}{2}})) \geq 1-O(m^{-1}) P ( ∥ K ( θ ) − K ∞ ∥ ≤ O (( Δ ϕ − 2 + m l 2 1 l ) κ ϕ 2 m − 2 1 )) ≥ 1 − O ( m − 1 ) Δ ϕ = b 2 a 2 + b 2 \Delta_\phi = \frac{b^2}{a^2+b^2} Δ ϕ = a 2 + b 2 b 2 κ ϕ = ∣ a ∣ + ∣ b ∣ a 2 + b 2 \kappa_\phi = \frac{|a| + |b|}{\sqrt{a^2 + b^2}} κ ϕ = a 2 + b 2 ∣ a ∣ + ∣ b ∣ Δ ϕ = 1 \Delta_\phi=1 Δ ϕ = 1 κ ϕ = 1 \kappa_\phi=1 κ ϕ = 1 Δ ϕ = 1 2 \Delta_\phi=\frac{1}{2} Δ ϕ = 2 1 κ ϕ = 2 \kappa_\phi=\sqrt{2} κ ϕ = 2
引用
@article{arxiv.2501.14724,
title = {MLPs at the EOC: Concentration of the NTK},
author = {Dávid Terjék and Diego González-Sánchez},
journal= {arXiv preprint arXiv:2501.14724},
year = {2025}
}
备注
36 pages, 1 figure