Related papers: CANFAR+Skytree: A Cloud Computing and Data Mining …
We have designed and implemented a novel way to process wide-field astronomical data within a distributed environment of hardware resources and humanpower. The system is characterized by integration of archiving, calibration, and…
The new NASA Astrophysics Data System (ADS) is designed with a serviceoriented architecture (SOA) that consists of multiple customized Apache Solr search engine instances plus a collection of microservices, containerized using Docker, and…
The Square Kilometre Array (SKA) will operate one of the world's largest continuous scientific data systems, sustaining petascale imaging under strict power envelopes. Current radio-interferometric pipelines typically achieve only 4-14% of…
Increasing amounts of data from varied sources, particularly in the fields of machine learning and graph analytics, are causing storage requirements to grow rapidly. A variety of technologies exist for storing and sharing these data,…
The data volumes stored in telescope archives is constantly increasing due to the development and improvements in the instrumentation. Often the archives need to be stored over a distributed storage architecture, provided by independent…
Scientific communities naturally tend to organize around data ecosystems created by the combination of their observational devices, their data repositories, and the workflows essential to carry their research from observation to discovery.…
SkyPortal is an open-source software package designed to efficiently discover interesting transients, manage follow-up, perform characterization, and visualize the results. By enabling fast access to archival and catalog data,…
Smartphones have exploded in popularity in recent years, becoming ever more sophisticated and capable. As a result, developers worldwide are building increasingly complex applications that require ever increasing amounts of computational…
Astrophysics has become a domain extremely rich of scientific data. Data mining tools are needed for information extraction from such large datasets. This asks for an approach to data management emphasizing the efficiency and simplicity of…
Analyzing large-scale scientific datasets presents substantial challenges due to their sheer volume, structural complexity, and the need for specialized domain knowledge. Automation tools, such as PandasAI, typically require full data…
The Virtual Research Environment is an analysis platform developed at CERN serving the needs of scientific communities involved in European Projects. Its scope is to facilitate the development of end-to-end physics workflows, providing…
Data access and interoperability module connects the observation proposals, data, virtual machines and software. According to the unique identifier of PI (principal investigator), an email address or an internal ID, data can be collected by…
For the successful development of the astrophysics and, accordingly, for obtaining more complete knowledge of the Universe, it is extremely important to combine and comprehensively analyze information of various types (e.g., about charged…
Astronomy is entering a new era as multiple, large area, digital sky surveys are in production. The resulting datasets are truly remarkable in their own right; however, a revolutionary step arises in the aggregation of complimentary…
As astronomy enters the petascale data era, astronomers are faced with new challenges relating to storage, access and management of data. A shift from the traditional approach of combining data and analysis at the desktop to the use of…
Scientific applications in HPC environment are more com-plex and more data-intensive nowadays. Scientists usually rely on workflow system to manage the complexity: simply define multiple processing steps into a single script and let the…
The Astrophysics Source Code Library (ASCL) contains 3000 metadata records about astrophysics research software and serves primarily as a registry of software, though it also can and does accept code deposit. Though the ASCL was started in…
I/O performance is crucial to efficiency in data-intensive scientific computing; but tuning large-scale storage systems is complex, costly, and notoriously manpower-intensive, making it inaccessible for most domain scientists. To address…
In recent years, cloud storage technology has been widely used in many fields such as education, business, medical and more because of its convenience and low cost. With the widespread applications of cloud storage technology, data access…
Modern applications span multiple clouds to reduce costs, avoid vendor lock-in, and leverage low-availability resources in another cloud. However, standard object stores operate within a single cloud, forcing users to manually manage data…