Related papers: Patterns of i.i.d. Sequences and Their Entropy - P…
In two previous experiments we investigated the neural precursors of subjects' "free" choices for one of two options (pressing one of two buttons, and choosing between adding and subtracting numbers). In these experiments the distribution…
This paper shows that the probability that the error exponent of a given code randomly generated from a pairwise independent ensemble being smaller than a lower bound on the typical random-coding exponent tends to zero as the codeword…
First principle network models are crucial to make sense of the intricate topology of real complex networks. While modeling efforts have been quite successful in undirected networks, generative models for networks with asymmetric…
Given a sequence composed of a limit number of characters, we try to "read" it as a "text". This involves to segment the sequence into "words". The difficulty is to distinguish good segmentation from enormous number of random ones.Aiming at…
An index of uniformity is developed as an alternative to the maximum-entropy principle for selecting continuous, differentiable probability distributions $\mathcal{P}$ subject to constraints $C$. The uniformity index developed in this paper…
We address this work to investigate symbolic sequences with long-range correlations by using computational simulation. We analyze sequences with two, three and four symbols that could be repeated $l$ times, with the probability distribution…
Bounds on information combining are entropic inequalities that determine how the information, or entropy, of a set of random variables can change when they are combined in certain prescribed ways. Such bounds play an important role in…
The phenomenon of entropy concentration provides strong support for the maximum entropy method, MaxEnt, for inferring a probability vector from information in the form of constraints. Here we extend this phenomenon, in a discrete setting,…
The ordinal patterns of a fixed number of consecutive values in a time series is the spatial ordering of these values. Counting how often a specific ordinal pattern occurs in a time series provides important insights into the properties of…
Discovering causal direction from temporal observational data is particularly challenging for symbolic sequences, where functional models and noise assumptions are often unavailable. We propose a novel \emph{Dictionary Based Pattern Entropy…
This article investigates integer sequences that partition the sequence into blocks of various lengths - irregular arrays. The main result of the article is explicit formulas for numbering of irregular arrays. A generalization of Cantor…
We study the notion of approximate entropy within the framework of network theory. Approximate entropy is an uncertainty measure originally proposed in the context of dynamical systems and time series. We firstly define a purely structural…
Conditional differential entropy provides an intuitive measure for relatively ranking time-series complexity by quantifying uncertainty in future observations given past context. However, its direct computation for high-dimensional…
With the rise of neural models across the field of information retrieval, numerous publications have incrementally pushed the envelope of performance for a multitude of IR tasks. However, these networks often sample data in random order,…
Biological sequences do not come at random. Instead, they appear with particular frequencies that reflect properties of the associated system or phenomenon. Knowing how biological sequences are distributed in sequence space is thus a…
Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations. This block structure has two seemingly contradictory properties: on the…
Cages, defined as regular graphs with minimum number of nodes for a given girth, are well-studied in graph theory. Trapping sets are graphical structures responsible for error floor of low-density parity-check (LDPC) codes, and are well…
Sequences have become first class citizens in supervised learning thanks to the resurgence of recurrent neural networks. Many complex tasks that require mapping from or to a sequence of observations can now be formulated with the…
An effective way to scale up test-time compute of large language models is to sample multiple responses and then select the best one, as in Grok Heavy and Gemini Deep Think. Existing selection methods often rely on external reward models,…
We introduce a family of scale-invariant entropy statistics derived from logarithmically aggregated distance distributions of point processes, with prime numbers serving as a motivating example. The construction associates to each finite…