Related papers: Designing a Linearized Potential Function in Neura…
In this paper, we investigate the partition inequality, joint convexity, and Pinsker's inequality, for a divergence that generalizes the Tsallis Relative Entropy and Kullback-Leibler divergence. The generalized divergence is defined in…
The Shannon entropy is a widely used summary statistic, for example, network traffic measurement, anomaly detection, neural computations, spike trains, etc. This study focuses on estimating Shannon entropy of data streams. It is known that…
The construction of efficient and effective decision trees remains a key topic in machine learning because of their simplicity and flexibility. A lot of heuristic algorithms have been proposed to construct near-optimal decision trees. ID3,…
Recent research suggests that looped Transformers have superior reasoning capabilities compared to standard deep architectures. Current approaches to training single-head looped architectures on benchmark tasks frequently fail or yield…
The lack of mathematical tractability of Deep Neural Networks (DNNs) has hindered progress towards having a unified convergence analysis of training algorithms, in the general setting. We propose a unified optimization framework for…
Algorithmic entropy and Shannon entropy are two conceptually different information measures, as the former is based on size of programs and the later in probability distributions. However, it is known that, for any recursive probability…
A well-known result across information theory, machine learning, and statistical physics shows that the maximum entropy distribution under a mean constraint has an exponential form called the Gibbs-Boltzmann distribution. This is used for…
In this paper, we investigate new procedures for statistical testing based on Tsallis entropy, a parametric generalization of Shannon entropy. Focusing on multivariate generalized Gaussian and $q$-Gaussian distributions, we develop…
Learning-to-optimize leverages machine learning to accelerate optimization algorithms. While empirical results show tremendous improvements compared to classical optimization algorithms, theoretical guarantees are mostly lacking, such that…
Characterising the capacity region for a network can be extremely difficult. Even with independent sources, determining the capacity region can be as hard as the open problem of characterising all information inequalities. The majority of…
Pattern learning in an important problem in Natural Language Processing (NLP). Some exhaustive pattern learning (EPL) methods (Bod, 1992) were proved to be flawed (Johnson, 2002), while similar algorithms (Och and Ney, 2004) showed great…
We propose a general method for optimization with semi-infinite constraints that involve a linear combination of functions, focusing on the case of the exponential function. Each function is lower and upper bounded on sub-intervals by…
The entropy error function has been widely used in neural networks. Nevertheless, the network training based on this error function generally leads to a slow convergence rate, and can easily be trapped in a local minimum or even with the…
The success of deep convolutional neural network (CNN) in computer vision especially image classification problems requests a new information theory for function of image, instead of image itself. In this article, after establishing a deep…
In many applications, the probability density function is subject to experimental errors. In this work the continuos dependence of a class of generalized entropies on the experimental errors is studied. This class includes the C. Shannon,…
Fueled by the expressive power of deep neural networks, normalizing flows have achieved spectacular success in generative modeling, or learning to draw new samples from a distribution given a finite dataset of training samples. Normalizing…
The maximum entropy principle advocates to evaluate events' probabilities using a distribution that maximizes entropy among those that satisfy certain expectations' constraints. Such principle can be generalized for arbitrary decision…
We shall show that a two-parameter extended entropy function is characterized by a functional equation. As a corollary of this result, we obtain that the Tsallis entropy function is characterized by a functional equation, which is a…
Tsallis' non-extensive entropy is extended to incorporate the dependence on affinities between the microstates of a system. At the core of our construction of the extended entropy ($\mathcal{H}$) is the concept of the effective number of…
In this current work, we propose a Max Min approach for approximating functions using exponential neural network operators. We extend this framework to develop the Max Min Kantorovich-type exponential neural network operators and…