相关论文: Soft-to-Hard Routing in Sparse Mixture-of-Experts …

In this manuscript, we consider the problems of channel assignment in wireless networks and data migration in heterogeneous storage systems. We show that a soft edge coloring approach to both problems gives rigorous approximation…

Mixture-of-Experts (MoE) has become a cornerstone in recent state-of-the-art large language models (LLMs). Traditionally, MoE relies on $\mathrm{Softmax}$ as the router score function to aggregate expert output, a designed choice that has…

We propose DropMax, a stochastic version of softmax classifier which at each iteration drops non-target classes according to dropout probabilities adaptively decided for each instance. Specifically, we overlay binary masking variables over…

A novel multiphysics-decision tree learning algorithm is presented for (1) estimating transport properties in the variably saturated subsurface governed by explicitly coupled equations for water, heat, and solute transport; and (2)…

In this article a unified approach to iterative soft-thresholding algorithms for the solution of linear operator equations in infinite dimensional Hilbert spaces is presented. We formulate the algorithm in the framework of generalized…

Direct numerical simulations of a low-pressure turbine with roughness elements distributed over the blade surface have been performed. A series of fifteen cases with varying roughness heights and streamwise wavenumbers are introduced to…

We analyze the achievable rate in interference-free wireless networks with physical layer fading channels and orthogonal multiple access. As a starting point, the point-to-point channel is considered. We find the optimal physical and…

The development of randomized algorithms for numerical linear algebra, e.g. for computing approximate QR and SVD factorizations, has recently become an intense area of research. This paper studies one of the most frequently discussed…

The thin interface limit aims at minimizing the effects arising from a numerical interface thickness, inherent in diffuse interface models of solidification and microstructure evolution such as the phase field model. While the original…

In this study, we systematically evaluate the impact of common design choices in Mixture of Experts (MoEs) on validation performance, uncovering distinct influences at token and sequence levels. We also present empirical evidence showing…

Common measures of neural representational (dis)similarity are designed to be insensitive to rotations and reflections of the neural activation space. Motivated by the premise that the tuning of individual units may be important, there has…

To aid in prediction of turbulent boundary layer flows over rough surfaces, a new model is proposed to estimate hydrodynamic roughness based solely on geometric surface information. The model is based on a fluid-mechanics motivated…

We study the statistical properties of learning to defer (L2D) to multiple experts. In particular, we address the open problems of deriving a consistent surrogate loss, confidence calibration, and principled ensembling of experts. Firstly,…

Parameter Efficient Tuning has been an prominent approach to adapt the Large Language Model to downstream tasks. Most previous works considers adding the dense trainable parameters, where all parameters are used to adapt certain task. We…

In this paper, we introduce a novel dynamic expert selection framework for Mixture of Experts (MoE) models, aiming to enhance computational efficiency and model performance by adjusting the number of activated experts based on input…

Distance distributions are a key building block in stochastic geometry modelling of wireless networks and in many other fields in mathematics and science. In this paper, we propose a novel framework for analytically computing the closed…

Backpressure-based adaptive routing algorithms where each packet is routed along a possibly different path have been extensively studied in the literature. However, such algorithms typically result in poor delay performance and involve high…

In this paper, we formulate the problem of inferring a Finite Mixture Model from discrete data as an optimal transport problem with entropic regularization of parameter $\lambda\geq 0$. Our method unifies hard and soft clustering, the…

We consider the problem of inferring the mobility map, which is the distribution of the building occupants at each timestamp, from the temperatures of the rooms. We also want to explore the effects of noise in the temperature measurement,…

We consider the high-dimensional linear regression model and assume that a fraction of the measurements are altered by an adversary with complete knowledge of the data and the underlying distribution. We are interested in a scenario where…