Related papers: Two-Scale Latent Dynamics for Recurrent-Depth Tran…

Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work,…

Discrete diffusion models have recently become competitive with autoregressive models for language modeling, even outperforming them on reasoning tasks requiring planning and global coherence, but they require more computation at inference…

This paper studies convergence properties of inexact iterative solution schemes for bilevel optimization problems. Bilevel optimization problems emerge in control-aware design optimization, where the system design parameters are optimized…

Stochastic gradient-based optimisation for discrete latent variable models is challenging due to the high variance of gradients. We introduce a variance reduction technique for score function estimators that makes use of double control…

We consider small-data, large-scale decision problems in which a firm must make many operational decisions simultaneously (e.g., across a large product portfolio) while observing only a few, potentially noisy, data points per instance.…

The solution of time-dependent hyperbolic conservation laws on cut cell meshes causes the small cell problem: standard schemes are not stable on the arbitrarily small cut cells if an explicit time stepping scheme is used and the time step…

Fine-tuning provides an effective means to specialize pre-trained models for various downstream tasks. However, fine-tuning often incurs high memory overhead, especially for large transformer-based models, such as LLMs. While existing…

During time evolution of many-body systems entanglement grows rapidly, limiting exact simulations to small-scale systems or small timescales. Quantum information tends however to flow towards larger scales without returning to local scales,…

Understanding the timescales associated with relaxation to equilibrium in closed quantum many-body systems is one of the central focuses in the study of their non-equilibrium dynamics. At late times, these relaxation processes exhibit…

We investigate the system size scaling of the net defect number created by a rapid quench in a second-order quantum phase transition from an O(N) symmetric state to a phase of broken symmetry. Using a controlled mean-field expansion for…

DETR-like methods have significantly increased detection performance in an end-to-end manner. The mainstream two-stage frameworks of them perform dense self-attention and select a fraction of queries for sparse cross-attention, which is…

Macroscopic properties of heterogeneous media are frequently modelled by regular lattice models, which are based on a relatively small basic cluster of lattice sites. Here, we extend one of such models to any cluster's size kxk. We also…

In this paper, we study the evolution of tokens through the depth of encoder-only transformer models at inference time by modeling them as a system of particles interacting in a mean-field way and studying the corresponding dynamics. More…

The dynamics of small-scale structures in free-surface turbulence is crucial to large-scale phenomena in natural and industrial environments. Here we conduct experiments on the quasi-flat free surface of a zero-mean-flow turbulent water…

Diffusion Transformers have recently demonstrated unprecedented generative capabilities for various tasks. The encouraging results, however, come with the cost of slow inference, since each denoising step requires inference on a transformer…

Several recent works demonstrate that transformers can implement algorithms like gradient descent. By a careful construction of weights, these works show that multiple layers of transformers are expressive enough to simulate iterations of…

We show that dynamical quantum phase transitions (DQPTs) in the quench dynamics of two-dimensional topological systems can be characterized by a dynamical topological invariant defined along an appropriately chosen closed contour in…

We study an array of two-level systems arranged on a lattice and illuminated by an external plane wave which drives a dipolar transition between the two energy levels. In this set up, the two-level systems are coupled by dipolar…

Large Language Models (LLMs) are trained with next-token prediction, implemented in autoregressive Transformers via causal masking for parallelism. This creates a subtle misalignment: residual connections tie activations to the current…

In quantum computing the decoherence time of the qubits determines the computation time available and this time is very limited when using current hardware. In this paper we minimize the execution time (the depth) for a class of circuits…