Related papers: Re-evaluating scaling methods for distributed para…
Computing systems have undergone several inflexion points - while Moore's law guided the semiconductor industry to cram more and more transistors and logic into the same volume, the limits of instruction-level parallelism (ILP) and the end…
Cloud Computing is a paradigm of both parallel processing and distributed computing. It offers computing facilities as a utility service in pay as par use manner. Virtualization, self service provisioning, elasticity and pay per use are the…
We demonstrate that conventional artificial deep neural networks operating near the phase boundary of the signal propagation dynamics, also known as the edge of chaos, exhibit universal scaling laws of absorbing phase transitions in…
Statistical laws describe regular patterns observed in diverse scientific domains, ranging from the magnitude of earthquakes (Gutenberg-Richter law) and metabolic rates in organisms (Kleiber's law), to the frequency distribution of words in…
This report provides an introduction to algorithms for fundamental linear algebra problems on various parallel computer architectures, with the emphasis on distributed-memory MIMD machines. To illustrate the basic concepts and key issues,…
The efficient parallel execution of complex computations requires balancing the workload across processors while minimizing the communication between them. This inherent trade-off is often captured by graph partitioning or DAG scheduling…
Cloud computing refers to maximizing efficiency by sharing computational and storage resources, while data-parallel systems exploit the resources available in the cloud to perform parallel transformations over large amounts of data. In the…
The area of online machine learning in big data streams covers algorithms that are (1) distributed and (2) work from data streams with only a limited possibility to store past data. The first requirement mostly concerns software…
While machine learning has advanced through massive parallelization, we identify a critical blind spot: some problems are fundamentally sequential. These "inherently serial" problems-from mathematical reasoning to physical simulations to…
In large storage systems, files are often coded across several servers to improve reliability and retrieval speed. We study load balancing under the Batch Sampling routing scheme for a network of $n$ servers storing a set of files using the…
Parallel computing is very important to accelerate the performance of software systems. Additionally, considering that a recurring challenge is to process high data volumes continuously, stream processing emerged as a paradigm and software…
Prior work on Automatically Scalable Computation (ASC) suggests that it is possible to parallelize sequential computation by building a model of whole-program execution, using that model to predict future computations, and then…
The evolution of distributed architectures and programming paradigms for performance-oriented program development, challenge the state-of-the-art technology for performance tools. The area of high performance computing is rapidly expanding…
With the analysis of the hierarchical scheme, the potential influence of the pre-constant in deriving scaling laws is exposed. It is found that a modified hierarchical scheme can achieve a throughput arbitrarily times higher than the…
The alternating direction method of multipliers (ADMM) has been recognized as a versatile approach for solving modern large-scale machine learning and signal processing problems efficiently. When the data size and/or the problem dimension…
Based on our previous work on algebraic laws for true concurrency, we design a structured parallel programming language for true concurrency called PPL. Different to most programming languages, PPL has an explicit parallel operator as an…
Neural scaling laws aim to characterize how out-of-sample error behaves as a function of model and training dataset size. Such scaling laws guide allocation of a computational resources between model and data processing to minimize error.…
Parallelization has become a cornerstone of modern computing, influencing everything from high performance supercomputers to everyday mobile devices. This paper presents a comprehensive guide on the fundamentals of parallelization that…
The article is devoted to the problem of applying the maximum principle for finding optimal control parameters in simulation tasks of interest for a variety of engineering and industrial systems and processes. Especially important is the…
A set of general allometric scaling laws is derived for different systems represented by tree networks. The formulation postulates self-similar networks with an arbitrary number of branches developed in each generation, and with an…