Related papers: Halting Time is Predictable for Large Models: A Un…
Large language models exhibit surprising emergent generalization properties, yet also struggle on many simple reasoning tasks such as arithmetic and parity. This raises the question of if and when Transformer models can learn the true…
In this work, we study the generalization capability of algorithms from an information-theoretic perspective. It has been shown that the expected generalization error of an algorithm is bounded from above by a function of the relative…
This study examines the generalization ability of algorithm performance prediction models across various benchmark suites. Comparing the statistical similarity between the problem collections with the accuracy of performance prediction…
Universality is a key hypothesis in mechanistic interpretability -- that different models learn similar features and circuits when trained on similar tasks. In this work, we study the universality hypothesis by examining how small neural…
Virtually all machine learning tasks are characterized using some form of loss function, and "good performance" is typically stated in terms of a sufficiently small average loss, taken over the random draw of test data. While optimizing for…
A prediction interval covers a future observation from a random process in repeated sampling, and is typically constructed by identifying a pivotal quantity that is also an ancillary statistic. Analogously, a tolerance interval covers a…
The majority of machine learning methods can be regarded as the minimization of an unavailable risk function. To optimize the latter, given samples provided in a streaming fashion, we define a general stochastic Newton algorithm and its…
This paper reports, by way of introduction, on the advances made by our group and the broader signal processing community on the concept of sample abundance; a phenomenon that naturally arises in one-bit and few-bit signal processing…
Stationarity is a very general, qualitative assumption, that can be assessed on the basis of application specifics. It is thus a rather attractive assumption to base statistical analysis on, especially for problems for which less general…
The unreduced, universally nonperturbative analysis of arbitrary interaction process, described by a quite general equation, provides the truly complete, "dynamically multivalued" general solution that leads to dynamically derived,…
In the last decades, many efforts have focused on analyzing typical-case hardness in optimization and inference problems. Some recent work has pointed out that polynomial algorithms exist, running with a time that grows more than linearly…
This article investigates emergence and complexity in complex systems that can share information on a network. To this end, we use a theoretical approach from information theory, computability theory, and complex networks. One key studied…
Permutation entropy measures the complexity of deterministic time series via a data symbolic quantization consisting of rank vectors called ordinal patterns or just permutations. The reasons for the increasing popularity of this entropy in…
Traditional algorithm analysis treats all basic operations as equally costly, which hides significant differences in time, energy consumption, and cost between different types of computations on modern processors. We propose a…
The true process that generated data cannot be determined when multiple explanations are possible. Prediction requires a model of the probability that a process, chosen randomly from the set of candidate explanations, generates some future…
This paper concerns the construction of tests for universal hypothesis testing problems, in which the alternate hypothesis is poorly modeled and the observation space is large. The mismatched universal test is a feature-based technique for…
In this paper, a simulation-based method for the analysis and design of abstracted models for a stochastic hybrid system is proposed. The accuracy of a model is evaluated in terms of its capability to reproduce the system output for all the…
As has long been known to computer scientists, the performance of probabilistic algorithms characterized by relatively large runtime fluctuations can be improved by applying a restart, i.e., episodic interruption of a randomized…
Deep neural networks often contain far more parameters than training examples, yet they still manage to generalize well in practice. Classical complexity measures such as VC-dimension or PAC-Bayes bounds usually become vacuous in this…
We present a new algorithm for general reinforcement learning where the true environment is known to belong to a finite class of N arbitrary models. The algorithm is shown to be near-optimal for all but O(N log^2 N) time-steps with high…