Related papers: Phase Transitions in Knowledge Compilation: an Exp…
General-purpose large language models (LLMs) are increasingly deployed in verticals such as telecommunications, where adaptation is hindered by scarce, low-information-density corpora and tight mobile/edge constraints. We propose Data…
We study theoretical limits of \emph{descending} phase retrieval algorithms. Utilizing \emph{Random duality theory} (RDT) we develop a generic program that allows statistical characterization of various algorithmic performance metrics.…
Continual learning refers to a dynamical framework in which a model receives a stream of non-stationary data over time and must adapt to new data while preserving previously acquired knowledge. Unluckily, neural networks fail to meet these…
An important challenge in statistical analysis concerns the control of the finite sample bias of estimators. For example, the maximum likelihood estimator has a bias that can result in a significant inferential loss. This problem is…
Phase transitions have recently been formulated in the time domain of quantum many-body systems, a phenomenon dubbed dynamical quantum phase transitions (DQPTs), whose phenomenology is often divided in two types. One refers to distinct…
We report an analytic and numerical study of a phase transition in a P problem (the assignment problem) that separates two phases whose representatives are the simple matching problem (an easy P problem) and the traveling salesman problem…
Transformer-based language models have demonstrated impressive capabilities across a range of complex reasoning tasks. Prior theoretical work exploring the expressive power of transformers has shown that they can efficiently perform…
In the past few years, transformer-based pre-trained language models have achieved astounding success in both industry and academia. However, the large model size and high run-time latency are serious impediments to applying them in…
Several query evaluation tasks can be done via knowledge compilation: the query result is compiled as a lineage circuit from which the answer can be determined. For such tasks, it is important to leverage some width parameters of the…
Consider a graph where each of the $n$ nodes is either in state $\mathcal{R}$ or $\mathcal{B}$. Herein, we analyze the \emph{synchronous $k$-Majority dynamics}, where in each discrete-time round nodes simultaneously sample $k$ neighbors…
Statistical inference problems arising within signal processing, data mining, and machine learning naturally give rise to hard combinatorial optimization problems. These problems become intractable when the dimensionality of the data is…
Random constraint satisfaction problems play an important role in computer science and combinatorics. For example, they provide challenging benchmark instances for algorithms and they have been harnessed in probabilistic constructions of…
Large language models (LLMs) have significantly advanced various natural language processing tasks, but deploying them remains computationally expensive. Knowledge distillation (KD) is a promising solution, enabling the transfer of…
In this conference proceeding, I discuss in detail the deconfinement to quark matter that takes place at large densities and/or temperatures. The first-order phase transition that is assumed to appear beyond a critical point gives rise to…
Motivated by novel results in the theory of correlated sequences, we analyze the dynamics of random walks with long-term memory (binary chains with long-range correlations). In our model, the probability for a unit bit in a binary string…
Multicanonical ensemble simulations for the simulation of first-order phase transitions suffer from exponential slowing down. Monte Carlo autocorrelation times diverge exponentially with free energy barriers $\Delta F$, which in $L^d$ boxes…
Leveraging knowledge from multiple tasks through introducing a small number of task specific parameters into each transformer layer, also known as adapters, receives much attention recently. However, adding an extra fusion layer to…
We consider the problem of change-point estimation of the instantaneous phase of an observed time series. Such change points, or phase shifts, can be markers of information transfer in complex systems; their analysis occurring in geology,…
Knowledge distillation (KD) is a widely used technique to transfer knowledge from a large teacher network to a smaller student model. Traditional KD uses a fixed balancing factor alpha as a hyperparameter to combine the hard-label…
We show new limits on the efficiency of using current techniques to make exact probabilistic inference for large classes of natural problems. In particular we show new lower bounds on knowledge compilation to SDD and DNNF forms. We give…