Related papers: MuCon: Clipped Muon Updates for LLM Training
We present a method for solving implicit (factored) Markov decision processes (MDPs) with very large state spaces. We introduce a property of state space partitions which we call epsilon-homogeneity. Intuitively, an epsilon-homogeneous…
The Muon optimizer has demonstrated strong empirical performance in pre-training large language models by performing matrix-level gradient (or momentum) orthogonalization in each layer independently. In this work, we propose TEON, a…
In this work, we consider rank-one adaptations $X_{new} = X+ab^T$ of a given matrix $X\in \mathbb{R}^{n\times p}$ with known matrix factorization $X = UW$, where $U\in\mathbb{R}^{n\times p}$ is column-orthogonal, i.e. $U^TU=I$. Arguably the…
U-Clip is a simple amendment to gradient clipping that can be applied to any iterative gradient optimization algorithm. Like regular clipping, U-Clip involves using gradients that are clipped to a prescribed size (e.g. with component wise…
An arbitrary $m\times n$ Boolean matrix $M$ can be decomposed {\em exactly} as $M =U\circ V$, where $U$ (resp. $V$) is an $m\times k$ (resp. $k\times n$) Boolean matrix and $\circ$ denotes the Boolean matrix multiplication operator. We…
Spectral optimizers such as Muon have recently shown strong empirical performance in large-scale language model training, but the source and extent of their advantage remain poorly understood. We study this question through the linear…
We present experimental work on a primal-dual framework simultaneously approximating maximum cut and weighted fractional cut-covering instances. In this primal-dual framework, we solve a semidefinite programming (SDP) relaxation to either…
We propose new primal-dual decomposition algorithms for solving systems of inclusions involving sums of linearly composed maximally monotone operators. The principal innovation in these algorithms is that they are block-iterative in the…
We present a simple method for the extraction of corrections for bias in the measurement of the momentum of muons in hadron collider experiments. Such bias can originate from a variety of sources such as detector misalignment, software…
Post-training compression of Transformer models commonly relies on truncated singular value decomposition (SVD). However, enforcing a single shared subspace can degrade accuracy even at moderate compression. Sparse dictionary learning…
We propose a new method for low-rank approximation of Moore-Penrose pseudoinverses (MPPs) of large-scale matrices using tensor networks. The computed pseudoinverses can be useful for solving or preconditioning of large-scale overdetermined…
A self-consistent version of the left-right (LR) symmetric model is used to examine tree- as well as one-loop level radiative corrections to the muon decay. It is shown that constraints on the heavy sector of the model parameters are…
Recently, prompt learning has garnered considerable attention for its success in various Vision-Language (VL) tasks. However, existing prompt-based models are primarily focused on studying prompt generation and prompt strategies with…
We propose a unitary precoding scheme, namely polar-precoding, to improve the performance of polar-coded MIMO systems. In contrast to the traditional design of MIMO precoding criteria, the proposed polar-precoding scheme relies on the…
Machine unlearning (MU) aims to remove the influence of specific data from a trained model. However, approximate unlearning methods, often formulated as a single-objective optimization (SOO) problem, face a critical trade-off between…
Several recent publications report advances in training optimal decision trees (ODT) using mixed-integer programs (MIP), due to algorithmic advances in integer programming and a growing interest in addressing the inherent suboptimality of…
Given a matrix $X\in \mathbb{R}^{m\times n}_+$ with non-negative entries, the cone factorization problem over a cone $\mathcal{K}\subseteq \mathbb{R}^k$ concerns computing $\{ a_1,\ldots, a_{m} \} \subseteq \mathcal{K}$ and $\{ b_1,\ldots,…
The singlet capture rate $\Lambda_S$ for the semileptonic weak process $\mu+p \to n+\nu_\mu$ has been measured in the MuCap experiment. The novel experimental technique is based on stopping muons in an active target, consisting of a time…
Subtraction schemes provide a systematic way to compute fully-differential cross sections beyond the leading order in the strong coupling constant. These methods make singular real-emission corrections integrable in phase space by the…
We study the implicit bias of momentum-based optimizers on smooth homogeneous models. We show that \textit{momentum steepest descent} algorithms like Muon (spectral norm), MomentumGD ($\ell_2$ norm), and Signum ($\ell_\infty$ norm) are…