Related papers: AXS: A framework for fast astronomical data proces…
Most of the popular Big Data analytics tools evolved to adapt their working environment to extract valuable information from a vast amount of unstructured data. The ability of data mining techniques to filter this helpful information from…
Data analytic applications built upon big data processing frameworks such as Apache Spark are an important class of applications. Many of these applications are not latency-sensitive and thus can run as batch jobs in data centers. By…
With the rapid growth of Next Generation Sequencing (NGS) technologies, large amounts of "omics" data are daily collected and need to be processed. Indexing and compressing large sequences datasets are some of the most important tasks in…
Growing data volumes and velocities are driving exciting new methods across the sciences in which data analytics and machine learning are increasingly intertwined with research. These new methods require new approaches for scientific…
This document was created in order to study the algorithms for the categorization of phrases and rank them using the facilities provided by the framework Apache Spark. Starting from the study illustrated in the publication "Classifying…
In today's society, location-based services are widely used which collect a huge amount of human trajectories. Analyzing semantic meanings of these trajectories can benefit numerous real-world applications, such as product advertisement,…
The AXIS Community Science Book represents the collective effort of 592 scientists worldwide to define the transformative science enabled by the Advanced X-ray Imaging Satellite (AXIS), a next-generation X-ray mission selected by NASA's…
Spatial data fusion is a bottleneck when it meets the scale of 10 billion records. Cross-matching celestial catalogs is just one example of this. To challenge this, we present a framework that enables efficient cross-matching using Learned…
Aims: ARCiS, a novel code for the analysis of exoplanet transmission and emission spectra is presented. The aim of the modelling framework is to provide a tool able to link observations to physical models of exoplanet atmospheres. Methods:…
Data collection for scientific applications is increasing exponentially and is forecasted to soon reach peta- and exabyte scales. Applications which process and analyze scientific data must be scalable and focus on execution performance to…
As the demand of astronomical observation rising, the telescope systems are becoming more and more complex. Thus, the observatory control software needs to be more intelligent, they have to control each instrument inside the observatory,…
Since its debut by John Skilling in 2004, nested sampling has proven a valuable tool to the scientist, providing hypothesis evidence calculations and parameter inference for complicated posterior distributions, particularly in the field of…
Non-linear spectral dimensionality reduction methods, such as Isomap, remain important technique for learning manifolds. However, due to computational complexity, exact manifold learning using Isomap is currently impossible from large-scale…
Motivation: Alignment-free distance and similarity functions (AF functions, for short) are a well established alternative to two and multiple sequence alignments for many genomic, metagenomic and epigenomic tasks. Due to data-intensive…
Data skipping reduces I/O for SQL queries by skipping over irrelevant data objects (files) based on their metadata. We extend this notion by allowing developers to define their own data skipping metadata types and indexes using a flexible…
Due to the significant importance of Big Data analysis, especially in business-related topics such as improving services, finding potential customers, and selecting practical approaches to manage income and expenses, many companies attempt…
New radio interferometers such as MeerKAT, SKA, ngVLA, and DSA-2000 drive advancements in software for two key reasons. First, handling the vast data from these instruments requires subdivision and multi-node processing. Second, their…
To obtain the best possible scientific result, astronomers must understand the properties of the available instrumentation well. This is important both when designing new instruments and when using existing instruments close to the limits…
We present the first public data release of the VISTA EXtension to Auxiliary Surveys (VEXAS), comprising of 9 cross-matched multi-wavelength photometric catalogs where each object has a match in at least two surveys. We aim at a spatial…
The volume of data generated by modern astronomical telescopes is extremely large and rapidly growing. However, current high-performance data processing architectures/frameworks are not well suited for astronomers because of their…