Related papers: Addressing NameNode Scalability Issue in Hadoop Di…
MapReduce (MR) is the most popular solution to build applications for large-scale data processing. These applications are often deployed on large clusters of commodity machines, where failures happen constantly due to bugs, hardware…
Monte Carlo simulations employed for the analysis of portfolios of catastrophic risk process large volumes of data. Often times these simulations are not performed in real-time scenarios as they are slow and consume large data. Such…
HPC environments have traditionally been designed to meet the compute demand of scientific applications and data has only been a second order concern. With science moving toward data-driven discoveries relying more on correlations in data…
The moving computation on the edge or near to data is the new trend that can break the bandwidth wall and to unleash the power of next generation NVM or SCM memory. File system is the important OS subsystem that plays the role of mediator…
This report evaluates the new analytical capabilities of DataStax Enterprise (DSE) [1] through the use of standard Hadoop workloads. In particular, we run experiments with CPU and I/O bound micro-benchmarks as well as OLAP-style analytical…
Distributed hash table (DHT) is the foundation of many widely used storage systems, for its prominent features of high scalability and load balancing. Recently, DHT-based systems have been deployed for the Internet-of-Things (IoT)…
Software-defined networks (SDNs) are a huge evolution in simplifying implementation and network operation which have reduced costs and made the network programmable. Although SDNs are a suitable option for solving some of the previous…
High-level I/O libraries, such as HDF5 and PnetCDF, are commonly used by large-scale scientific applications to perform I/O tasks in parallel. These I/O libraries store the metadata such as data types and dimensionality along with the raw…
Edge computing is the practice of placing computing resources at the edges of the Internet in close proximity to devices and information sources. This, much like a cache on a CPU, increases bandwidth and reduces latency for applications but…
With the advent of cloud computing, thousands of machines are connected and managed collectively. This era is confronted with a new challenge: performance variability, primarily caused by large-scale management issues such as hardware…
The latest trends in high-performance computing systems show an increasing demand on the use of a large scale multicore systems in a efficient way, so that high compute-intensive applications can be executed reasonably well. However, the…
High-performance computing (HPC) is essential for tackling complex computational problems across various domains. As the scale and complexity of HPC applications continue to grow, the need for scalable systems and software architectures…
We consider a wireless device-to-device (D2D) network where $n$ nodes are uniformly distributed at random over the network area. We let each node with storage capacity $M$ cache files from a library of size $m \geq M$. Each node in the…
Spark is an in-memory analytics platform that targets commodity server environments today. It relies on the Hadoop Distributed File System (HDFS) to persist intermediate checkpoint states and final processing results. In Spark, immutable…
When dealing with massive data sorting, we usually use Hadoop which is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. A common approach in implement of…
The emerging hybrid DRAM-NVM architecture is challenging the existing memory management mechanism in operating system. In this paper, we introduce memos, which can schedule memory resources over the entire memory hierarchy including cache,…
In distributed storage systems (DSSs), the optimal tradeoff between node storage and repair bandwidth is an important issue for designing distributed coding strategies to ensure large scale data reliability. The capacity of DSSs is obtained…
This paper presents a stream-oriented architecture for structuring cluster applications. Clusters that run applications based on this architecture can scale to tenths of thousands of nodes with significantly less performance loss or…
A distributed quantum storage code maps a quantum message to N storage nodes, of arbitrary specified sizes, such that the stored message is robust to an arbitrary specified set of erasure patterns. The sizes of the storage nodes, and…
In this paper, we consider the algorithmic task of content replication and request routing in a distributed caching system consisting of a central server and a large number of caches, each with limited storage and service capabilities. We…