Related papers: Phase Transitions in Knowledge Compilation: an Exp…
This work presents a novel four-stage open-domain QA pipeline R2-D2 (Rank twice, reaD twice). The pipeline is composed of a retriever, passage reranker, extractive reader, generative reader and a mechanism that aggregates the final…
Analyzing a Feature Model (FM) and reasoning on the corresponding configuration space is a central task in Software Product Line (SPL) engineering. Problems such as deciding the satisfiability of the FM and eliminating inconsistent parts of…
We use the quantum Fisher information (QFI) to diagnose a dynamical phase transition (DPT) in a closed quantum system, which is usually defined in terms of non-analytic behaviour of a time-averaged order parameter. Employing the…
The organization of high-dimensional probability spaces is a fundamental problem at the intersection of statistical physics and information theory. Here, we analyze the distributions populating level surfaces of the probability simplex…
Recent advances in knowledge distillation (KD) have enabled smaller student models to approach the performance of larger teacher models. However, popular methods such as supervised KD and on-policy KD, are adversely impacted by the…
Despite substantial progress in multilingual extractive Question Answering (QA), models with high and uniformly distributed performance across languages remain challenging, especially for languages with limited resources. We study…
We discuss how phase-transitions may be detected in computationally hard problems in the context of Anytime Algorithms. Treating the computational time, value and utility functions involved in the search results in analogy with quantities…
In speech enhancement, knowledge distillation (KD) compresses models by transferring a high-capacity teacher's knowledge to a compact student. However, conventional KD methods train the student to mimic the teacher's output entirely, which…
The variety of the phase transitions in Induced QCD are studied. Depending upon the parameters in the scalar field potential, there could be infinite number of fixed points, with different critical behavior. The integral equation for the…
The concept of knowledge distillation (KD) describes the training of a student model from a teacher model and is a widely adopted technique in deep learning. However, it is still not clear how and why distillation works. Previous studies…
In this paper we study random linear systems with $k$ variables per equation over the finite field GF(2), or equivalently $k$-XOR-CNF formulas. In a previous paper Creignou and Daud\'e proved that the phase transition for the consistency…
Large Language Models (LLMs) have showcased exceptional capabilities in various domains, attracting significant interest from both academia and industry. Despite their impressive performance, the substantial size and computational demands…
Knowledge distillation (KD) is a widely adopted approach for compressing large neural networks by transferring knowledge from a large teacher model to a smaller student model. In the context of large language models, token level KD,…
The typical complexity of Constraint Satisfaction Problems (CSPs) can be investigated by means of random ensembles of instances. The latter exhibit many threshold phenomena besides their satisfiability phase transition, in particular a…
Phase transitions generically occur in random matrix models as the parameters in the joint probability distribution of the random variables are varied. They affect all main features of the theory and the interpretation of statistical models…
The existence and search for thermodynamic phase transitions is of unfading interest. In this paper, we present numerical evidence of dynamical phase transitions occurring in boundary driven systems with a constrained integrated current. It…
Direct Feedback Alignment (DFA) is emerging as an efficient and biologically plausible alternative to the ubiquitous backpropagation algorithm for training deep neural networks. Despite relying on random feedback weights for the backward…
Knowledge distillation (KD) has been extensively employed to transfer the knowledge from a large teacher model to the smaller students, where the parameters of the teacher are fixed (or partially) during training. Recent studies show that…
Equilibrium statistical physics is applied to layered neural networks with differentiable activation functions. A first analysis of off-line learning in soft-committee machines with a finite number (K) of hidden units learning a perfectly…
Recent developments in the unsupervised domain adaptation (UDA) enable the unsupervised machine learning (ML) prediction for target data, thus this will accelerate real world applications with ML models such as image recognition tasks in…