Related papers: The Transpension Type: Technical Report
We study in-context learning problems where a Transformer is pretrained on tasks drawn from a mixture distribution $\pi=\sum_{\alpha\in\mathcal{A}} \lambda_{\alpha} \pi_{\alpha}$, called the pretraining prior, in which each mixture…
The past year has witnessed the rapid development of applying the Transformer module to vision problems. While some researchers have demonstrated that Transformer-based models enjoy a favorable ability of fitting data, there are still…
Modern deep neural networks display striking examples of rich internal computational structure. Uncovering principles governing the development of such structure is a priority for the science of deep learning. In this paper, we explore the…
The recently introduced model of representations has been defined and motivated somewhat ex-nihilo. In this document, I will show that representations are related to a more ''classical'' model through a 2-adjunction. The target model is…
Following the types-as-sets paradigm, we present a mechanized embedding of dependent function types with a hierarchy of universes into schematic first-order logic with equality, with axiom schemas of Tarski-Grothendieck set theory. We carry…
The similarity of feature representations plays a pivotal role in the success of problems related to domain adaptation. Feature similarity includes both the invariance of marginal distributions and the closeness of conditional distributions…
Foundation models have achieved great advances in multi-task learning with a unified interface of unimodal and multimodal tasks. However, the potential of such multi-task learners has not been exploited during transfer learning. In this…
This work involves the usage of various NLP models to predict the winner of a particular judgment by the means of text extraction and summarization from a judgment document. These documents are useful when it comes to legal proceedings. One…
We study the interaction of structural subtyping with parametric polymorphism and recursively defined type constructors. Although structural subtyping is undecidable in this setting, we describe a notion of parametricity for type…
We show in many cases the existence of adjoints to extension of scalars on categories of motivic nature, in the framework of field extensions. This is to be contrasted with the more classical situation where one deals with a finite type…
This paper improves the session typing theory to support the modelling and verification of processes that implement federated learning protocols. To this end, we build upon the asynchronous ``bottom-up'' session typing approach by adding…
The increasing of pre-trained models has significantly facilitated the performance on limited data tasks with transfer learning. However, progress on transfer learning mainly focuses on optimizing the weights of pre-trained models, which…
We develop new techniques for constructing model structures from a given class of cofibrations, together with a class of fibrant objects and a choice of weak equivalences between them. As a special case, we obtain a more flexible version of…
The interpretation of feature importance in machine learning models is challenging when features are dependent. Permutation feature importance (PFI) ignores such dependencies, which can cause misleading interpretations due to extrapolation.…
By theorems of Carlson and Renaudin, the theory of $(\infty,1)$-categories embeds in that of prederivators. The purpose of this paper is to give a two-fold answer to the inverse problem: understanding which prederivators model…
Most categorical models for dependent types have traditionally been heavily set based: contexts form a category, and for each we have a set of types in said context -- and for each type a set of terms of said type. This is the case for…
Transfer learning seeks to improve the generalization performance of a target task by exploiting the knowledge learned from a related source task. Central questions include deciding what information one should transfer and when transfer can…
We develop here the algebra of the differential field of transseries and of related valued differential fields. This book contains in particular our recently obtained decisive positive results on the model theory of these structures.
Tensor transpose is a higher order generalization of matrix transpose. In this paper, we use permutations and symmetry group to define? the tensor transpose. Then we discuss the classification and composition of tensor transposes.…
Transfer learning methods endeavor to leverage relevant knowledge from existing source pre-trained models or datasets to solve downstream target tasks. With the increase in the scale and quantity of available pre-trained models nowadays, it…