分布式、并行与集群计算

The common assumption in on-device AI is that GPUs, with their superior parallel processing, always provide the best performance for large language model (LLM) inference. In this work, we challenge this notion by empirically demonstrating…

Large language model (LLM) training is one of the most demanding distributed computations today, often requiring thousands of GPUs with frequent synchronization across machines. Such a workload pattern makes it susceptible to stragglers,…

We present a deterministic parallel multilevel algorithm for balanced hypergraph partitioning that matches the state of the art for non-deterministic algorithms. Deterministic parallel algorithms produce the same result in each invocation,…

Text-to-image generation using diffusion models has gained increasing popularity due to their ability to produce high-quality, realistic images based on text prompts. However, efficiently serving these models is challenging due to their…

This paper proposes a novel approach to address the challenges of deploying complex robotic software in large-scale systems, i.e., Centralized Nonlinear Model Predictive Controllers (CNMPCs) for multi-agent systems. The proposed approach is…

The concept of distributed consensus originated in the 1970s and gained widespread attention following Leslie Lamport's influential publication on the Byzantine Generals Problem in the 1980s. Over the past five decades, distributed…

Smart-city services are typically developed as closed systems within each city's vertical, communicating and interacting with cloud services while remaining isolated within each provider's domain. With the emergence of 5G private domains…

Scaling multi-dimensional transformers to long sequences is indispensable across various domains. However, the challenges of large memory requirements and slow speeds of such sequences necessitate sequence parallelism. All existing…

Programming modern high-performance computing systems is challenging due to the need to efficiently program GPUs and accelerators and to handle data movement between nodes. The C++ language has been continuously enhanced in recent years…

Heterogeneous multi-core architectures combine on a single chip a few large, general-purpose host cores, optimized for single-thread performance, with (many) clusters of small, specialized, energy-efficient accelerator cores for…

Pipeline parallelism is a crucial paradigm for large-scale model training. However, imbalances in memory footprint across stages can lead to significant GPU memory wastage, limiting the model sizes that pipeline parallelism can effectively…

We present the first sublinear-in-$n$ round algorithm for sampling an approximately uniform spanning tree of an $n$-vertex graph in the CongestedClique model of distributed computing. In particular, our algorithm requires…

The exponential growth of big data has transformed how large organisations leverage information to drive innovation, optimise processes, and maintain competitive advantages. However, managing and extracting insights from vast, heterogeneous…

In the distributed localization problem (DLP), $n$ anonymous robots (agents) $a_0, a_1, ..., a_{n-1}$ begin at arbitrary positions $p_0, ..., p_{n-1}$ in $S$, where $S$ is an Euclidean space. The primary goal in DLP is for agents to reach a…

In this paper we describe a prototype of a blockchain-in-a-box system which allows users to easily bootstrap the whole Ethereum Proof-of-Work (PoW) network running on multiple Raspberry Pi nodes - an inexpensive modular computers. Users are…

This paper describes a new asynchronous algorithm and implementation for the problem of k-mer counting (KC), which concerns quantifying the frequency of length k substrings in a DNA sequence. This operation is common to many computational…

Job Shop Scheduling (JSS) is one of the most studied combinatorial optimization problems. It involves scheduling a set of jobs with predefined processing constraints on a set of machines to achieve a desired objective, such as minimizing…

In this paper we analyze some of the bottlenecks in the execution pipeline of Solana's Agave validator client, focusing on RAM and program cache usage under mainnet conditions. Through a series of controlled experiments, we measure the…

Accurate determination of the performance of parallel GPU code typically requires execution-time profiling on target hardware -- an increasingly prohibitive step due to limited access to high-end GPUs. This paper explores whether Large…

Resource allocation in High Performance Computing (HPC) environments presents a complex and multifaceted challenge for job scheduling algorithms. Beyond the efficient allocation of system resources, schedulers must account for and optimize…