Related papers: Variance Is Not Importance: Structural Analysis of…
This paper investigates voltage stability in inverter-based power systems concerning fold and saddle-node bifurcations. An analytical expression is derived for the sensitivity of the stability margin using the normal vector to the…
A detailed high pressure study is carried out on 1T$^\prime$ MoTe$_{2}$ using X-ray diffraction(XRD) and Raman spectroscopy measurements upto about 30.5 GPa. High pressure XRD measurements show no structural transition. All the lattice…
Transformers have achieved extraordinary success in modern machine learning due to their excellent ability to handle sequential data, especially in next-token prediction (NTP) tasks. However, the theoretical understanding of their…
The class of complex random vectors whose covariance matrix is linearly parameterized by a basis of Hermitian Toeplitz (HT) matrices is considered, and the maximum compression ratios that preserve all second-order information are derived…
Neural network (NN) designed for challenging machine learning tasks is in general a highly nonlinear mapping that contains massive variational parameters. High complexity of NN, if unbounded or unconstrained, might unpredictably cause…
How do transformer language models represent magnitude? Recent work disagrees: some find logarithmic spacing, others linear encoding, others per-digit circular representations. We apply the formal tools of psychophysics to resolve this.…
Large Language Models (LLMs) are not only resource-intensive to train but even more costly to deploy in production. Therefore, recent work has attempted to prune blocks of LLMs based on cheap proxies for estimating block importance,…
We consider the possibility that the primordial curvature perturbation is direction-dependent. To first order this is parameterised by a quadrupolar modulation of the power spectrum and results in statistical anisotropy of the CMB, which…
Despite the popularity of transformers in practice, their architectures are empirically designed and neither mathematically justified nor interpretable. Moreover, as indicated by many empirical studies, some components of transformer…
We introduce token-consistent stochastic layers in vision transformers, without causing any severe drop in performance. The added stochasticity improves network calibration, robustness and strengthens privacy. We use linear layers with…
We experimentally and numerically study the precise role of geometry for the mechanics of biholar metamaterials, quasi-2D slabs of rubber patterned by circular holes of two alternating sizes. We recently showed how the response to uniaxial…
Recent works have indicated redundancy across transformer blocks, prompting the research of depth compression to prune less crucial blocks. However, current ways of entire-block pruning suffer from risks of discarding meaningful cues…
Fine-tuning adapts pretrained networks to new objectives. Whether the resulting depth profile of representational change reflects an intrinsic property of the model or the magnitude of gradient flow has not been tested directly. We measure…
With the proliferation of mobile devices and the Internet of Things, deep learning models are increasingly deployed on devices with limited computing resources and memory, and are exposed to the threat of adversarial noise. Learning deep…
Compression has been a critical lens to understand the success of Transformers. In the past, we have typically taken the target distribution as a criterion to evaluate a model's compression performance. Nevertheless,it often remains…
A recent body of work has demonstrated that Transformer embeddings can be linearly decomposed into well-defined sums of factors, that can in turn be related to specific network inputs or components. There is however still a dearth of work…
A widely cited result by Dong et al. (2021) showed that Transformers built from self-attention alone, without skip connections or feed-forward layers, suffer from rapid rank collapse: all token representations converge to a single…
Pre-trained Transformer models like T5 and BART have advanced the state of the art on a wide range of text generation tasks. Compressing these models into smaller ones has become critically important for practical use. Common neural network…
Resistive memories are outstanding electron devices that have displayed a large potential in a plethora of applications such as nonvolatile data storage, neuromorphic computing, hardware cryptography, etc. Their fabrication control and…
Large-eddy simulations of a flat-plate boundary layer, without a leading edge, subject to multiple levels of incoming free stream turbulence are considered in the present work. Within an input-output model where non-linear terms of the…