Related papers: Scalability in Computing and Robotics
The integration of more renewable energy sources into the power system is presenting system operators with various challenges. At the distribution system level, voltage magnitudes that violate operating limits near large photovoltaic…
Downstream scaling laws aim to predict task performance at larger scales from the model's performance at smaller scales. Whether such prediction should be possible is unclear: some works discover clear linear scaling trends after simple…
Recent work on neural scaling laws demonstrates that model performance scales predictably with compute budget, model size, and dataset size. In this work, we develop scaling laws based on problem complexity. We analyze two fundamental…
Compositional understanding is crucial for human intelligence, yet it remains unclear whether contemporary vision models exhibit it. The dominant machine learning paradigm is built on the premise that scaling data and model sizes will…
Scaling laws illuminate Nature's fundamental biological principles and guide bioinspired materials and structural designs. In simple cases they are based on the fundamental principle that all laws of nature remain unchanged (i.e.,…
We study the dynamics of a system composed of interacting units each with a complex internal structure comprising many subunits. We consider the case in which each subunit grows in a multiplicative manner. We propose a model for such…
In collective systems, the available agents are a limited resource that must be allocated among tasks to maximize collective performance. Computing the optimal allocation of several agents to numerous tasks through a brute-force approach…
The objective of statistical physics is to understand macroscopic behavior of a many-body system from the interactions of the constituents of that system. When many-body systems reach critical states, simple universal and scaling behaviors…
High capacity and scalable memory systems play a vital role in enabling our desktops, smartphones, and pervasive technologies like Internet of Things (IoT). Unfortunately, memory systems are becoming increasingly prone to faults. This is…
In this work and the supporting Parts II [2] and III [3], we provide a rather detailed analysis of the stability and performance of asynchronous strategies for solving distributed optimization and adaptation problems over networks. We…
The paper highlights that the cooperation of the components of the computing systems receives even more focus in the coming age of exascale computing. It discovers that inherent performance limitations exist and identifies the major…
Large scale systems are forecasted to greatly impact our future lives thanks to their wide ranging applications including cooperative robotics, mobility on demand, resource allocation, supply chain management. While technological…
Model generalization of the underlying dynamics is critical for achieving data efficiency when learning for robot control. This paper proposes a novel approach for learning dynamics leveraging the symmetry in the underlying robotic system,…
Scaling laws, a defining feature of deep learning, reveal a striking power-law improvement in model performance with increasing dataset and model size. Yet, their mathematical origins, especially the scaling exponent, have remained elusive.…
Growth patterns of complex systems predict how they change in sizes, numbers, masses, etc. Understanding growth is important, especially for many biological, ecological, urban, and socioeconomic systems. One noteworthy growth behavior is…
Large number of cores and hardware resource sharing are two characteristics on multicore processors, which bring new challenges for the design of operating systems. How to locate and analyze the speedup restrictive factors in operating…
Autonomous neural systems must efficiently process information in a wide range of novel environments, which may have very different statistical properties. We consider the problem of how to optimally distribute receptors along a…
Neural scaling laws, which in some domains can predict the performance of large neural networks as a function of model, data, and compute scale, are the cornerstone of building foundation models in Natural Language Processing and Computer…
Synchronization is the major obstacle to scalability in distributed computing. Concurrent operations on the shared data engage in synchronization when they encounter a \emph{conflict}, i.e., their effects depend on the order in which they…
As large-scale AI models expand, training becomes costlier and sustaining progress grows harder. Classical scaling laws (e.g., Kaplan et al. (2020), Hoffmann et al. (2022)) predict training loss from a static compute budget yet neglect time…