Related papers: Transformer-based prediction of two-dimensional ma…

Bearing fault detection is a critical task in predictive maintenance, where accurate and timely fault identification can prevent costly downtime and equipment damage. Traditional attention mechanisms in Transformer neural networks often…

Structure factors obtained from diffraction experiments are one of the most important quantities for characterizing the electronic and structural properties of materials. Methods for calculating this quantity from plane-wave density…

Strain governs not only the mechanical response of materials but also their electronic, optical, and catalytic properties. For this reason, the measurement of the 3D strain field is crucial for a detailed understanding and for further…

Electronic susceptibilities are a very popular tool to study electronic and magnetic properties of materials, both in experiment and theory. Unfortunately, the numerical evaluation of even the bare susceptibility, which depends on the…

We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through which different components of Transformer, such as the…

Vision Transformers has demonstrated competitive performance on computer vision tasks benefiting from their ability to capture long-range dependencies with multi-head self-attention modules and multi-layer perceptron. However, calculating…

The combination of deep learning and ab initio materials calculations is emerging as a trending frontier of materials science research, with deep-learning density functional theory (DFT) electronic structure being particularly promising. In…

We study theoretically the structural and electronic response of layered bulk black phosphorus to in-layer strain. Ab initio density functional theory (DFT) calculations reveal that the strain energy and interlayer spacing display a strong…

Recently, Transformer-based models for long sequence time series forecasting have demonstrated promising results. The self-attention mechanism as the core component of these Transformer-based models exhibits great potential in capturing…

The Transformer architecture has significantly advanced deep learning, particularly in natural language processing, by effectively managing long-range dependencies. However, as the demand for understanding complex relationships grows,…

The electronic properties of two-dimensional (2D) materials depend sensitively on the underlying atomic arrangement down to the monolayer level. Here we present a novel strategy for the determination of the band gap and complex dielectric…

The paradigm of Transformers using the self-attention mechanism has manifested its advantage in learning graph-structured data. Yet, Graph Transformers are capable of modeling full range dependencies but are often deficient in extracting…

Multi-scale simulations of nonlinear heterogeneous materials and composites are challenging due to the prohibitive computational costs of high-fidelity simulations. Recently, machine learning (ML) based approaches have emerged as promising…

Transformer-based models have demonstrated considerable potential for source code modeling tasks in software engineering. However, they are limited by their dependence solely on automatic self-attention weight learning mechanisms. Previous…

In deep learning theory, the covariance matrix of the representations serves as a proxy to examine the network's trainability. Motivated by the success of Transformers, we study the covariance matrix of a modified Softmax-based attention…

We present a rigorous analysis that combines theory, simulation, and experimental measurements to quantify the relationship between strain and bandgap in two dimensional gallium selenide (Ga$_2$Se$_2$). Experimentally, we transfer thin…

Large-scale foundation models for scientific machine learning adapt to physical settings unseen during training, such as zero-shot transfer between turbulent scales. This phenomenon, in-context learning, challenges conventional…

Transformers have recently gained attention in the computer vision domain due to their ability to model long-range dependencies. However, the self-attention mechanism, which is the core part of the Transformer model, usually suffers from…

We introduce Attention Free Transformer (AFT), an efficient variant of Transformers that eliminates the need for dot product self attention. In an AFT layer, the key and value are first combined with a set of learned position biases, the…

Strain engineering has emerged as a powerful tool to modify the optical and electronic properties of two-dimensional crystals. Here we perform a systematic study of strained semiconducting transition metal dichalcogenides. The effect of…