Related papers: Metric entropy of causal, discrete-time LTI system…
We investigate the impact of entropy change in deep learning systems by noise injection at different levels, including the embedding space and the image. The series of models that employ our methodology are collectively known as Noisy…
Imitation learning considerably simplifies policy synthesis compared to alternative approaches by exploiting access to expert demonstrations. For such imitation policies, errors away from the training samples are particularly critical. Even…
The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible while constrained to match empirically estimated feature expectations. However, in many real-world…
Despite growing interest in data stream mining the most successful incremental learners, such as VFDT, still use periodic recomputation to update attribute information gains and Gini indices. This note provides simple incremental formulas…
This work presents a novel means for understanding learning dynamics and scaling relations in neural networks. We show that certain measures on the spectrum of the empirical neural tangent kernel, specifically entropy and trace, yield…
This work investigates robust monotonic convergent iterative learning control (ILC) for uncertain linear systems in both time and frequency domains, and the ILC algorithm optimizing the convergence speed in terms of $l_{2}$ norm of error…
Recursive learning -- where models are trained on data generated by previous versions of themselves -- is increasingly common in large language models, autonomous agents, and self-supervised systems. However, standard performance metrics…
Recurrent neural networks (RNNs) process input text sequentially and model the conditional transition between word tokens. In contrast, the advantages of recursive networks include that they explicitly model the compositionality and the…
Attention mechanisms have been extensively employed in various applications, including time series modeling, owing to their capacity to capture intricate dependencies; however, their utility is often constrained by quadratic computational…
Much of the information the brain processes and stores is temporal in nature - a spoken word or a handwritten signature, for example, is defined by how it unfolds in time. However, it remains unclear how neural circuits encode complex…
We introduce a novel generalization of entropy and conditional entropy from which most definitions from the literature can be derived as particular cases. Within this general framework, we investigate the problem of designing…
We show that any matrix product state (MPS) can be exactly represented by a recurrent neural network (RNN) with a linear memory update. We generalize this RNN architecture to 2D lattices using a multilinear memory update. It supports…
Sequences and time-series often arise in robot tasks, e.g., in activity recognition and imitation learning. In recent years, deep neural networks (DNNs) have emerged as an effective data-driven methodology for processing sequences given…
This study proposes two straightforward yet effective approaches to reduce the required computational time of the training process for time-series modeling through a recurrent neural network (RNN) using multi-time-scale time-series data as…
We introduce a new concept of finite-time entropy which is a local version of the classical concept of metric entropy. Based on that, a finite-time version of Pesin's entropy formula and also an explicit formula of finite-time entropy for…
Recurrent neural networks (RNNs) are a widely used tool for modeling sequential data, yet they are often treated as inscrutable black boxes. Given a trained recurrent network, we would like to reverse engineer it--to obtain a quantitative,…
Classical neural network approximation results take the form: for every function $f$ and every error tolerance $\epsilon > 0$, one constructs a neural network whose architecture and weights depend on $\epsilon$. This paper introduces a…
Shannon entropy is not the only entropy that is relevant to machine-learning datasets, nor possibly even the most important one. Traditional entropies such as Shannon entropy capture information represented by elements' frequencies but not…
Several studies demonstrate that there are critical differences between real wireless networks and simulation models. This finding has permitted to extract spatial and temporal properties for links and to provide efficient methods as biased…
Post-processing of the raw bits produced by a true random number generator (TRNG) is always necessary when the entropy per bit is insufficient for security applications. In this paper, we derive a tight bound on the output min-entropy of…