Related papers: How to explain grokking
A geometric Brownian motion with delay is the solution of a stochastic differential equation where the drift and diffusion coefficient depend linearly on the past of the solution, i.e. a linear stochastic functional differential equation.…
For humans, language production and comprehension is sensitive to the hierarchical structure of sentences. In natural language processing, past work has questioned how effectively neural sequence models like transformers capture this…
Recent research on the grokking phenomenon has illuminated the intricacies of neural networks' training dynamics and their generalization behaviors. Grokking refers to a sharp rise of the network's generalization accuracy on the test set,…
Coarse graining (CG) is an important task for efficient modeling and simulation of complex multi-scale systems, such as the conformational dynamics of biomolecules. This work presents a projection-based coarse-graining formalism for general…
The Mori-Zwanzig formalism is applied to derive an equation for the evolution of linear observables of the overdamped Langevin equation. To illustrate the resulting equation and its use in deriving approximate models, a particular benchmark…
In physics, complex systems are often simplified into minimal, solvable models that retain only the core principles. In machine learning, layerwise linear models (e.g., linear neural networks) act as simplified representations of neural…
Explaining the predictions of neural black-box models is an important problem, especially when such models are used in applications where user trust is crucial. Estimating the influence of training examples on a learned neural model's…
Two different versions of relativistic Langevin equation in curved spacetime background are constructed, both are manifestly general covariant. It is argued that, from the observer's point of view, the version which takes the proper time of…
The motion of a particle under the influence of a dynamical disorder is described by the DLD model. One motivation is to understand the motion of an electron inside a metal; Another is to understand quantal Brownian motion. The overview is…
The generalization mystery in deep learning is the following: Why do over-parameterized neural networks trained with gradient descent (GD) generalize well on real datasets even though they are capable of fitting random datasets of…
We analyze the generalization gap (gap between the training and test errors) when training a potentially over-parametrized model using a Markovian stochastic training algorithm, initialized from some distribution $\theta_0 \sim p_0$. We…
Temporal knowledge graphs (TKGs) have shown promise for reasoning tasks by incorporating a temporal dimension to represent how facts evolve over time. However, existing TKG reasoning (TKGR) models lack explainability due to their black-box…
We present a derivation of a coarse-grained model from the Langevin dynamics. The focus is placed on the memory kernel function and the fluctuation-dissipation theorem. Also presented is an hierarchy of approximations for the memory and…
Grokking describes a delayed generalization phenomenon in which a neural network achieves perfect training accuracy long before validation accuracy improves, followed by an abrupt transition to strong generalization. Existing detection…
A data-driven ab initio generalized Langevin equation (AIGLE) approach is developed to learn and simulate high-dimensional, heterogeneous, coarse-grained conformational dynamics. Constrained by the fluctuation-dissipation theorem, the…
We construct a model of Brownian Motion on a pseudo-Riemannian manifold associated with general relativity. There are two aspects of the problem: The first is to define a sequence of stopping times associated with the Brownian "kicks" or…
A generalized Langevin equation is suggested to describe a system with memory($u(t,t') = \frac{1}{\Gamma (\nu )}(t - t')^\nu $) as well as with positive and negative damping. The equation can be transformed into the Fokker-Planck equation…
Financial market dynamics is rigorously studied via the exact generalized Langevin equation. Assuming market Brownian self-similarity, the market return rate memory and autocorrelation functions are derived, which exhibit an…
In this paper, we generalize the theory of Brownian motion and the Onsager-Machlup theory of fluctuations for spatially symmetric systems to equilibrium and nonequilibrium steady-state systems with a preferred spatial direction, due to an…
We study Brownian motion driven with both conservative and nonconservative external forces. By using the thermodynamic approach of the theory of Brownian motion we obtain the Fokker-Planck equation and derive expressions for the Fluctuation…