Related papers: Muown: Row-Norm Control for Muon Optimization
Shape optimization has been playing an important role in a large variety of engineering applications. Existing shape optimization methods are generally mesh-dependent and therefore encounter challenges due to mesh deformation. To overcome…
The past several decades have seen significant advancement in applications using cosmic-ray muons for tomography scanning of unknown objects. One of the most promising developments is the application of this technique in border security for…
We compute the dominant, logarithmically enhanced radiative corrections to the electron spectrum in the bound muon decay in the whole experimentally interesting range. The corrected spectrum fits well the TWIST results. The remaining…
Adam with decoupled weight decay, also known as AdamW, is widely acclaimed for its superior performance in language modeling tasks, surpassing Adam with $\ell_2$ regularization in terms of generalization and optimization. However, this…
A new muon beamline, muon science innovative channel (MuSIC), was set up at the Research Centre for Nuclear Physics (RCNP), Osaka University, in Osaka, Japan, using the 392 MeV proton beam impinging on a target. The production of an intense…
Several variations of adaptive first-order and second-order optimization methods have been proposed to accelerate and scale the training of large language models. The performance of these optimization routines is highly sensitive to the…
Cosmic muon imaging technology is increasingly being applied in various fields. However, simulating cosmic muons typically requires the rapid generation of a large number of muons and tracking their complex trajectories through intricate…
The muon has been instrumental in establishing the Standard Model of particle physics and continues to play a key role in exploring the nature of New Physics. A global program is underway to enhance the discovery potential of a wide range…
The weight decay regularization term is widely used during training to constrain expressivity, avoid overfitting, and improve generalization. Historically, this concept was borrowed from the SVM maximum margin principle and extended to…
Machine Unlearning (MU) aims at removing the influence of specific data from a pretrained model while preserving performance on the remaining data. In this work, a novel perspective for MU is presented upon low-dimensional feature…
We introduce Tune without Validation (Twin), a pipeline for tuning learning rate and weight decay without validation sets. We leverage a recent theoretical framework concerning learning phases in hypothesis space to devise a heuristic that…
Deep learning optimizers are optimization algorithms that enable deep neural networks to learn. The effectiveness of learning is highly dependent on the optimizer employed in the training process. Alongside the rapid advancement of deep…
Adaptive methods like Adam have become the $\textit{de facto}$ standard for large-scale vector and Euclidean optimization due to their coordinate-wise adaptation with a second-order nature. More recently, matrix-based spectral optimizers…
Generalisation of a deep neural network (DNN) is one major concern when employing the deep learning approach for solving practical problems. In this paper we propose a new technique, named approximated orthonormal normalisation (AON), to…
In this paper, we introduce weight prediction into the AdamW optimizer to boost its convergence when training the deep neural network (DNN) models. In particular, ahead of each mini-batch training, we predict the future weights according to…
We study adaptive learning rate scheduling for norm-constrained optimizers (e.g., Muon and Lion). We introduce a generalized smoothness assumption under which local curvature decreases with the suboptimality gap and empirically verify that…
Meson factories are powerful drivers of diverse physics programmes. With beam powers already in the MW-regime attention has to be turned to target and beam line design to further significantly increase surface muon rates available for…
We analyze Muon as originally proposed and used in practice -- using the momentum orthogonalization with a few Newton-Schulz steps. The prior theoretical results replace this key step in Muon with an exact SVD-based polar factor. We prove…
Scaling laws have made language-model performance predictable from model size, data, and compute, but they typically treat the optimizer as a fixed training detail. We show that this assumption misses a fundamental axis of representation…
Deep learning methods for material property prediction have been widely explored to advance materials discovery. However, the prevailing pre-train then fine-tune paradigm often fails to address the inherent diversity and disparity of…