Related papers: Scientific Computing Meets Big Data Technology: An…
We explore the trade-offs of performing linear algebra using Apache Spark, compared to traditional C and MPI implementations on HPC platforms. Spark is designed for data analytics on cluster computing platforms with access to local disks…
We present a science-driven discovery portal for all the ESA Astronomy Missions called ESA Sky that allow users to explore the multi-wavelength sky and to seamlessly retrieve science-ready data in all ESA Astronomy mission archives from a…
Innovative developments in data processing, archiving, analysis, and visualization are nowadays unavoidable to deal with the data deluge expected in next-generation facilities for radio astronomy, such as the Square Kilometre Array (SKA)…
The scientific community is presently witnessing an unprecedented growth in the quality and quantity of data sets coming from simulations and real-world experiments. To access effectively and extract the scientific content of such…
As data volumes grow across applications, analytics of large amounts of data is becoming increasingly important. Big data processing frameworks such as Apache Hadoop, Apache AsterixDB, and Apache Spark have been built to meet this demand. A…
The Square Kilometre Array (SKA) Observatory is gearing up the formal construction of its two radio interferometers in Australia and South Africa after the end of design and pre-construction phases. Agile methodologies, the Cloud native…
Healthcare data is a valuable resource for research, analysis, and decision-making in the medical field. However, healthcare data is often fragmented and distributed across various sources, making it challenging to combine and analyze…
scida is a Python package for reading and analyzing large scientific data sets with support for various cosmological and galaxy formation simulations out-of-the-box. Data access is provided through a hierarchical dictionary-like data…
Big data analytics requires high programmer productivity and high performance simultaneously on large-scale clusters. However, current big data analytics frameworks (e.g. Apache Spark) have prohibitive runtime overheads since they are…
The ongoing exponential growth of computational power, and the growth of the commercial High Performance Computing (HPC) industry, has led to a point where ten commercial systems currently exceed the performance of the highest-used HPC…
We present the results of our investigations into options for the computing platform for the imaging pipeline in the CHILES project, an ultra-deep HI pathfinder for the era of the Square Kilometre Array. CHILES pushes the current computing…
One of the major performance and scalability bottlenecks in large scientific applications is parallel reading and writing to supercomputer I/O systems. The usage of parallel file systems and consistency requirements of POSIX, that all the…
In the last decades, scientific software has graduated from a hidden side-product to a first-class member of the astrophysics literature. We aim to quantify the activity and impact of software development for astronomy, using a systematic…
ESASky is a science-driven discovery portal to explore the multi-wavelength sky and visualise and access multiple astronomical archive holdings. The tool is a web application that requires no prior knowledge of any of the missions involved…
Today's high-performance computing (HPC) systems are heavily instrumented, generating logs containing information about abnormal events, such as critical conditions, faults, errors and failures, system resource utilization, and about the…
The Apache Spark stack has enabled fast large-scale data processing. Despite a rich library of statistical models and inference algorithms, it does not give domain users the ability to develop their own models. The emergence of…
Apache Spark is a popular system aimed at the analysis of large data sets, but recent studies have shown that certain computations---in particular, many linear algebra computations that are the basis for solving common machine learning…
In the era of "big data" and with the advent of web 2.0 technologies, ESASky (http://sky.esa.int) aims at providing a modern and visual way to access astronomical science-ready data products and metadata. The main goal of the application is…
Visualizing and navigating through large astronomy images from a remote location with current astronomy display tools can be a frustrating experience in terms of speed and ergonomics, especially on mobile devices. In this paper, we present…
We present a case study of a cloud-based computational workflow for processing large astronomical data sets from the Murchison Widefield Array (MWA) cosmology experiment. Cloud computing is well-suited to large-scale, episodic computation…