Related papers: No Free Swap: Protocol-Dependent Layer Redundancy …

For years the model performance in machine learning obeyed a power-law relationship with the model size. For the consideration of parameter efficiency, recent studies focus on increasing model depth rather than width to achieve better…

Deploying transformer models in practice is challenging due to their inference cost, which scales quadratically with input sequence length. To address this, we present a novel Learned Token Pruning (LTP) method which adaptively removes…

The next generation wireless systems will face stringent new requirements, including ultra-low latency, high data rates and enhanced reliability. Large Intelligent Surfaces, is one proposed solution that has the potential to solve these…

Consider a MIMO interference channel whereby each transmitter and receiver are equipped with multiple antennas. The basic problem is to design optimal linear transceivers (or beamformers) that can maximize system throughput. The recent work…

Overparameterized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and question answering. These models contain hundreds of millions of…

This work considers load-balance control among the relays under the secure transmission protocol via relay cooperation in two-hop wireless networks without the information of both eavesdropper channels and locations. The available two-hop…

In response to the development of recent efficient dense layers, this paper shows that something as simple as replacing linear components in pointwise convolutions with structured linear decompositions also produces substantial gains in the…

An infinite hierarchy of layering transitions exists for model polymers in solution under poor solvent or low temperatures and near an attractive surface. A flat histogram stochastic growth algorithm known as FlatPERM has been used on a…

Learning, prediction, and compression are intimately connected: a model that accurately predicts the next symbol in a sequence can be coupled with a source coder to compress that sequence near its information-theoretic limit. When tokenized…

Although large language models (LLMs) have achieved remarkable success across various domains, their considerable scale necessitates substantial computational resources, posing significant challenges for deployment in resource-constrained…

Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often…

This work investigates distillation methods for large language models (LLMs) with the goal of developing compact models that preserve high performance. Several existing approaches are reviewed, with a discussion of their respective…

The possible paralelism existing between phase transitions and fracture in disordered materials, is discussed using the well-known Fiber Bundle Models and a probabilistic approach suited to smooth fluctuations near the critical point. Two…

Recently much attention has been paid to the study of the robustness of interdependent and multiplex networks and, in particular, networks of networks. The robustness of interdependent networks can be evaluated by the size of a mutually…

The rapid development in the performance of large language models (LLMs) is accompanied by the escalation of model size, leading to the increasing cost of model training and inference. Previous research has discovered that certain layers in…

There is evidence that transformers offer state-of-the-art recognition performance on tasks involving overhead imagery (e.g., satellite imagery). However, it is difficult to make unbiased empirical comparisons between competing deep…

Since its inception in "Attention Is All You Need", transformer architecture has led to revolutionary advancements in NLP. The attention layer within the transformer admits a sequence of input tokens $X$ and makes them interact through…

Recent work on permutation-based model merging has shown impressive low- or zero-barrier mode connectivity between models from completely different initializations. However, this line of work has not yet extended to the Transformer…

Transactional data may be represented as a bipartite graph $G:=(L \cup R, E)$, where $L$ denotes agents, $R$ denotes objects visible to many agents, and an edge in $E$ denotes an interaction between an agent and an object. Unsupervised…

From generation to generation there are increasing requirements for wireless standards both in terms of spectral and energy efficiency. While up to now the layered wireless transceiver architecture worked allowing for, e.g., separation of…