相关论文: Analysis of an information-theoretic model for com…
Kaplan et al. [2020] (`Kaplan') and Hoffmann et al. [2022] (`Chinchilla') studied the scaling behavior of transformers trained on next-token language prediction. These studies produced different estimates for how the number of parameters…
Understanding the innovation process, that is the underlying mechanisms through which novelties emerge, diffuse and trigger further novelties is undoubtedly of fundamental importance in many areas (biology, linguistics, social science and…
Zipf's law is found when the vocabulary of long written texts is ranked according to the frequency of word occurrences, establishing a power-law decay for the frequency vs rank relation. This law is a robust statistical property observed…
The minimum heat cost of computation is subject to bounds arising from Landauer's principle. Here, I derive bounds on finite modelling -- the production or anticipation of patterns (time-series data) -- by devices that model the pattern in…
The cost of communication is a substantial factor affecting the scalability of many distributed applications. Every message sent can incur a cost in storage, computation, energy and bandwidth. Consequently, reducing the communication costs…
The optimal solution of an inter-city passenger transport network has been studied using Zipf's law for the city populations and the Gravity law describing the fluxes of inter-city passenger traffic. Assuming a fixed value for the cost of…
In 1981 Hong and Kung proved a lower bound on the amount of communication needed to perform dense, matrix-multiplication using the conventional $O(n^3)$ algorithm, where the input matrices were too large to fit in the small, fast memory. In…
Zipf's law on word frequency is observed in English, French, Spanish, Italian, and so on, yet it does not hold for Chinese, Japanese or Korean characters. A model for writing process is proposed to explain the above difference, which takes…
Irreversible information processing cannot be carried out without some inevitable thermodynamical work cost. This fundamental restriction, known as Landauer's principle, is increasingly relevant today, as the energy dissipation of computing…
In this work we are concerned with the design of efficient mechanisms while eliciting limited information from the agents. First, we study the performance of sampling approximations in facility location games. Our key result is to show that…
We prove lower bounds for the direct sum problem for two-party bounded error randomised multiple-round communication protocols. Our proofs use the notion of information cost of a protocol, as defined by Chakrabarti, Shi, Wirth and Yao and…
One of the main focuses in distributed learning is communication efficiency, since model aggregation at each round of training can consist of millions to billions of parameters. Several model compression methods, such as gradient…
An important body of quantitative linguistics is constituted by a series of statistical laws about language usage. Despite the importance of these linguistic laws, some of them are poorly formulated, and, more importantly, there is no…
We study the communication complexity of computing functions $F:\{0,1\}^n\times \{0,1\}^n \rightarrow \{0,1\}$ in the memoryless communication model. Here, Alice is given $x\in \{0,1\}^n$, Bob is given $y\in \{0,1\}^n$ and their goal is to…
The communication complexity of a quantum channel is the minimal amount of classical communication required for classically simulating a process of state preparation, transmission through the channel and subsequent measurement. It…
In the optimal partial transport problem, one is asked to transport a fraction $0<m \leq \min\{||f||_{L^1}, ||g||_{L^1}\}$ of the mass of $f=f \chi_\Omega$ onto $g=g\chi_\Lambda$ while minimizing a transportation cost. If $f$ and $g$ are…
Natural languages are full of rules and exceptions. One of the most famous quantitative rules is Zipf's law which states that the frequency of occurrence of a word is approximately inversely proportional to its rank. Though this `law' of…
I propose a cheap-talk model in which the sender can use private messages and only cares about persuading a subset of her audience. For example, a candidate only needs to persuade a majority of the electorate in order to win an election. I…
We consider the issue of multiple agents learning to communicate through reinforcement learning within partially observable environments, with a focus on information asymmetry in the second part of our work. We provide a review of the…
The entanglement cost of a quantum channel is the minimal rate at which entanglement (between sender and receiver) is needed in order to simulate many copies of a quantum channel in the presence of free classical communication. In this…