Related papers: Space filling positionality and the Spiroformer
Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry…
The $sp(2M)$ invariant unfolded system is considered in the periodic twistor-like spinor space. Complete set of non-trivial charges corresponding to the global symmetry compatible with the periodicity conditions is constructed. Residual…
Transformers have significantly advanced the field of natural language processing, but comprehending their internal mechanisms remains a challenge. In this paper, we introduce a novel geometric perspective that elucidates the inner…
Data augmentation in feature space is effective to increase data diversity. Previous methods assume that different classes have the same covariance in their feature distributions. Thus, feature transform between different classes is…
In this work, we study theoretical models of \emph{programmable matter} systems. The systems under consideration consist of spherical modules, kept together by magnetic forces and able to perform two minimal mechanical operations (or…
Building on Schlessinger's work, we define a framework for studying geometric deformation problems which allows us to systematize the relationship between the local and global tangent and obstruction spaces of a deformation problem.…
Phase transitions mark qualitative reorganizations of collective behavior, yet identifying their boundaries remains challenging whenever analytic solutions are absent and conventional simulations fail. Here we introduce learnability as a…
With the development of the self-attention mechanism, the Transformer model has demonstrated its outstanding performance in the computer vision domain. However, the massive computation brought from the full attention mechanism became a…
When placed on an inclined plane, a perfect 2D disk or 3D sphere simply rolls down in a straight line under gravity. But how is the rolling affected if these shapes are irregular or random? Treating the terminal rolling speed as an order…
This paper studies how Transformer models with Rotary Position Embeddings (RoPE) develop emergent, wavelet-like properties that compensate for the positional encoding's theoretical limitations. Through an analysis spanning model scales,…
Vacuum polarization of a massive scalar field in the background of a two-dimensional version of a spinning cosmic string is investigated. It is shown that when the `radius of the universe' is such that spacetime is globally hyperbolic the…
Transformers are being used extensively across several sequence modeling tasks. Significant research effort has been devoted to experimentally probe the inner workings of Transformers. However, our conceptual and theoretical understanding…
Learning representations on large-sized graphs is a long-standing challenge due to the inter-dependence nature involved in massive data points. Transformers, as an emerging class of foundation encoders for graph-structured data, have shown…
We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein…
This paper investigates the rotational dynamics on the higher-order Poincar\'e sphere with the use of $q$-plate by exploring three key aspects: the topological condition, the global-local rotation, and the SU(2) polarization evolution on…
Mesh denoising, aimed at removing noise from input meshes while preserving their feature structures, is a practical yet challenging task. Despite the remarkable progress in learning-based mesh denoising methodologies in recent years, their…
In a previous work we showcased the factorization method to find the symmetries of superintegrable systems with spherical separability in flat spaces. Here we analyze the same problem, but in constant curvature spaces along the examples of…
Multi-label image classification is about predicting a set of class labels that can be considered as orderless sequential data. Transformers process the sequential data as a whole, therefore they are inherently good at set prediction. The…
Graph Transformers typically rely on explicit positional or structural encodings and dense global attention to incorporate graph topology. In this work, we show that neither is essential. We introduce HopFormer, a graph Transformer that…
We follow our general model in Ref. [3] and analyze the formation of retinotopic projections for the biologically relevant situation of spherical geometries. To this end we elaborate both a linear and a nonlinear synergetic analysis which…