Related papers: Data Mining the SDSS SkyServer Database
We present the Skysoft project (http://www.skysoft.org). Skysoft is a YAASD (Yet Another Astronomical Software Directory), but with a different overall approach. To be useful, Skysoft needs to be a long-lived project, setting little…
The Sloan Digital Sky Survey (SDSS) will carry out a digital photometric and spectroscopic survey over pi steradians in the northern Galactic cap. An array of CCD detectors used in drift-scan mode will image the sky in five passbands to a…
The NASA Astrophysics Data System (ADS) is the primary Digital Library portal for researchers in astronomy and astrophysics. Over the past 30 years, the ADS has gone from being an astronomy-focused bibliographic database to an open digital…
Within the big data tsunami, relational databases and SQL are still there and remain mandatory in most of cases for accessing data. On the one hand, SQL is easy-to-use by non specialists and allows to identify pertinent initial data at the…
Clustering is one of the major tasks in data mining. In the last few years, Clustering of spatial data has received a lot of research attention. Spatial databases are components of many advanced information systems like geographic…
Nowadays medium-large size astronomical projects have to face the management of a large amount of information and data. Dedicated data centres manage the collection of raw and processed data and consequently make them accessible, typically…
We present a new analysis of the Sloan Digital Sky Survey data aimed at producing a detailed map of the nearby (z < 0.5) universe. Using neural networks trained on the available spectroscopic base of knowledge we derived distance estimates…
The fourth generation of the Sloan Digital Sky Survey (SDSS-IV) began observations in July 2014. It pursues three core programs: APOGEE-2, MaNGA, and eBOSS. In addition, eBOSS contains two major subprograms: TDSS and SPIDERS. This paper…
We describe the design and implementation of a high performance cloud that we have used to archive, analyze and mine large distributed data sets. By a cloud, we mean an infrastructure that provides resources and/or services over the…
The Smithsonian/NASA Astrophysics Data System (ADS) provides a search system for the astronomy and physics scholarly literature. All major and many smaller astronomy journals that were published on paper have been scanned back to volume 1…
Skyline and ranking queries are two of the most used tools to manage large data sets. The former is based on non-dominance, while the latter on a scoring function. Despite their effectiveness, they have some drawbacks like the result size…
The Astrophysics Source Code Library (ASCL) contains 3000 metadata records about astrophysics research software and serves primarily as a registry of software, though it also can and does accept code deposit. Though the ASCL was started in…
Apache Spark is a Big Data framework for working on large distributed datasets. Although widely used in the industry, it remains rather limited in the academic community or often restricted to software engineers. The goal of this paper is…
This paper describes the Fifth Data Release (DR5) of the Sloan Digital Sky Survey (SDSS). DR5 includes all survey quality data taken through June 2005 and represents the completion of the SDSS-I project (whose successor, SDSS-II will…
This is a thought piece on data-intensive science requirements for databases and science centers. It argues that peta-scale datasets will be housed by science centers that provide substantial storage and processing for scientists who access…
Sky coverage is one of the most important pieces of information about astronomical observations. We discuss possible representations, and present algorithms to create and manipulate shapes consisting of generalized spherical polygons with…
Microservices have become a popular architectural style for data-driven applications, given their ability to functionally decompose an application into small and autonomous services to achieve scalability, strong isolation, and…
We present a spectroscopic catalog of 70,841 visually inspected M dwarfs from the seventh data release of the Sloan Digital Sky Survey (SDSS). For each spectrum, we provide measurements of the spectral type, a number of molecular bandheads,…
Scientific applications produce a huge amount of data, which imposes serious management and analysis challenges. In particular, limitations in current database management systems prevent their adoption in simulation applications, in which…
We present status and results of AstroGrid-D, a joint effort of astrophysicists and computer scientists to employ grid technology for scientific applications. AstroGrid-D provides access to a network of distributed machines with a set of…