Related papers: AXS: A framework for fast astronomical data proces…

As data volumes grow across applications, analytics of large amounts of data is becoming increasingly important. Big data processing frameworks such as Apache Hadoop, Apache AsterixDB, and Apache Spark have been built to meet this demand. A…

XAssist is a NASA AISR-funded project for the automation of X-ray astrophysics, with emphasis on galaxies. It is nearing completion of its initially funded effort, and is working well for Chandra and ROSAT HRI data. Initial support for…

Domain scientists increasingly develop Python scripts to analyze satellite imagery but they lack scalability to large-scale data. This paper demonstrates GRAIL, an agentic translation system that converts Python geospatial workflows into…

ArchNEMESIS is an open-source Python package developed for the analysis of remote sensing spectroscopic observations of planetary atmospheres. It is based on the widely used NEMESIS radiative transfer and retrieval tool, which has been…

We present a framework for a large-scale distributed eScience Artificial Intelligence search. Our approach is generic and can be used for many different problems. Unlike many other approaches, we do not require dedicated machines,…

Data processing pipelines represent an important slice of the astronomical software library that include chains of processes that transform raw data into valuable information via data reduction and analysis. In this work we present Corral,…

BigDatalog is an extension of Datalog that achieves performance and scalability on both Apache Spark and multicore systems to the point that its graph analytics outperform those written in GraphX. Looking back, we see how this realizes the…

Contemporary exoplanet and brown dwarf atmospheric research relies heavily on retrieval frameworks to recover thermal and chemical properties and perform model comparison in an observational data-driven approach. However, the computational…

Scientists increasingly rely on Python tools to perform scalable distributed memory array operations using rich, NumPy-like expressions. However, many of these tools rely on dynamic schedulers optimized for abstract task graphs, which often…

Iris is an extensible application that provides astronomers with a user-friendly interface capable of ingesting broad-band data from many different sources in order to build, explore, and model spectral energy distributions (SEDs). Iris…

We develop a software kit called "2MASS Catalog Server Kit" to easily construct a high-performance database server for the 2MASS Point Source Catalog (includes 470,992,970 objects) and several all-sky catalogs. Users can perform fast radial…

Rapid growth of datacenter (DC) scale, urgency of cost control, increasing workload diversity, and huge software investment protection place unprecedented demands on the operating system (OS) efficiency, scalability, performance isolation,…

The new NASA Astrophysics Data System (ADS) is designed with a serviceoriented architecture (SOA) that consists of multiple customized Apache Solr search engine instances plus a collection of microservices, containerized using Docker, and…

To fulfill the requirements of space object cataloging and enable automated intelligent responses to anomalous events, we designed a novel observation scheduling system named Sky Survey Schedule (SSS). This framework facilitates coordinated…

Scalable distributed dataflow systems have recently experienced widespread adoption, with commodity dataflow engines such as Hadoop and Spark, and even commodity SQL engines routinely supporting increasingly sophisticated analytics tasks…

While cluster computing frameworks are continuously evolving to provide real-time data analysis capabilities, Apache Spark has managed to be at the forefront of big data analytics for being a unified framework for both, batch and stream…

SDAMS is the ensemble of database + software packages aimed to the archiving, quick-look analysis, off-line analysis, network accessibility and plotting of the SPOrt produced data. Many of the aspects related to data archiving, analysis and…

funcX is a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. Unlike centralized FaaS systems, funcX decouples the cloud-hosted management functionality from…

SPARKX is an open-source Python package developed to analyze simulation data from heavy-ion collision experiments. By offering a comprehensive suite of tools, SPARKX simplifies data analysis workflows, supports multiple formats such as…

The proliferation of mobile devices, such as smartphones and Internet of Things (IoT) gadgets, results in the recent mobile big data (MBD) era. Collecting MBD is unprofitable unless suitable analytics and learning methods are utilized for…