Related papers: Architecture of Information
This paper expresses the structure of artificial neural network (ANN) as a functional form, using the activation integral concept derived from the activation function. In this way, the structure of ANN can be represented by a simple…
The AHP/ANP are multicriteria decision-making theories that deal with both hierarchic structures when the criteria are independent of the alternatives and with networks when there is any dependence within and between elements of the…
We explore the connection between deep learning and information theory through the paradigm of diffusion models. A diffusion model converts noise into structured data by reinstating, imperfectly, information that is erased when data was…
This paper revisits Brownian motion from the perspective of Information Theory, aiming to explore the connections between Information Theory, Thermodynamics, and Complex Science. First, we propose a single-particle discrete Brownian motion…
Artificial Neural Networks (ANNs) were devised as a tool for Artificial Intelligence design implementations. However, it was soon became obvious that they are unable to fulfill their duties. The fully autonomous way of ANNs working,…
Embedded systems acquire information about the real world from sensors and process it to make decisions and/or for transmission. In some situations, the relationship between the data and the decision is complex and/or the amount of data to…
With the rapid development of Deep Neural Networks (DNNs), various network models that show strong computing power and impressive expressive power are proposed. However, there is no comprehensive informational interpretation of DNNs from…
Understanding the inner workings of deep neural networks (DNNs) is essential to provide trustworthy artificial intelligence techniques for practical applications. Existing studies typically involve linking semantic concepts to units or…
Artificial Neural Networks(ANN) has been phenomenally successful on various pattern recognition tasks. However, the design of neural networks rely heavily on the experience and intuitions of individual developers. In this article, the…
Efficient representation learning is essential for optimal information storage and classification. However, it is frequently overlooked in artificial neural networks (ANNs). This neglect results in networks that can become overparameterized…
This work explores entropy analysis as a tool for probing information distribution within Transformer-based architectures. By quantifying token-level uncertainty and examining entropy patterns across different stages of processing, we aim…
Over the last few years, key architectural advances have been proposed for neural network interatomic potentials (NNIPs), such as incorporating message-passing networks, equivariance, or many-body expansion terms. Although modern NNIP…
For sensory networks, we determine the rate with which they acquire information about the changing external conditions. Comparing this rate with the thermodynamic entropy production that quantifies the cost of maintaining the network, we…
By 'informational entropy', we understand an inherent boundary to knowability, due e.g. to perceptual, theoretical, evidential or linguistic limits. In this paper, we discuss a logical framework in which this boundary is incorporated into…
The Cognitive Buffer Hypothesis (CBH) posits that larger brains evolved to enhance survival in changing conditions. However, larger brains also carry higher energy demands, imposing additional metabolic burdens. Alongside brain size, brain…
This chapter provides a comprehensive and self-contained discussion of the most recent developments of information theory of networks. Maximum entropy models of networks are the least biased ensembles enforcing a set of constraints and are…
The practice of scientific research is often thought of as individuals and small teams striving for disciplinary advances. Yet as a whole, this endeavor more closely resembles a complex system of natural computation, in which information is…
For real-world complex system constantly enduring perturbation, to achieve survival goal in changing yet unknown environments, the central problem is constantly adapting themself to external environments according to environmental feedback.…
Entropy and information provide natural measures of correlation among elements in a network. We construct here the information theoretic analog of connected correlation functions: irreducible $N$--point correlation is measured by a decrease…
I consider motivation and value-alignment in AI systems from the perspective of (constrained) entropy maximization. Though the structures encoding knowledge in any physical system can be understood as energetic constraints, only living…