Related papers: Astronomy in the Cloud: Using MapReduce for Image …
We tackle the problem of predicting the performance of MapReduce applications, designing accurate progress indicators that keep programmers informed on the percentage of completed computation time during the execution of a job. Through…
We describe the design and implementation of a high performance cloud that we have used to archive, analyze and mine large distributed data sets. By a cloud, we mean an infrastructure that provides resources and/or services over the…
This paper describes a new approach to the optimization of information extraction in multi-wavelength image cubes of cosmological fields. The objective is to create a framework for the automatic identification and tagging of sources…
Current ground-based submillimeter instruments (e.g. SCUBA-2, SHARC-2 and LABOCA) have hundreds to thousands of detectors, sampled at tens to hundreds of hertz, generating up to hundreds of gigabytes per night. Since noise is correlated…
In this study we present a two-step map/reduce framework to stitch satellite mosaic images. The proposed system enable recognition and extraction of objects whose parts falling in separate satellite mosaic images. However this is a time and…
Currently, the processing of scientific data in astroparticle physics is based on various distributed technologies, the most common of which are Grid and cloud computing. The most frequently discussed approaches are focused on large and…
Software Defined Networking (SDN) is a revolutionary network architecture that separates out network control functions from the underlying equipment and is an increasingly trend to help enterprises build more manageable data centers where…
Hadoop MapReduce is now a popular choice for performing large-scale data analytics. This technical report describes a detailed set of mathematical performance models for describing the execution of a MapReduce job on Hadoop. The models…
HOLODEC, an airborne cloud particle imager, captures holographic images of a fixed volume of cloud to characterize the types and sizes of cloud particles, such as water droplets and ice crystals. Cloud particle properties include position,…
Additive spatial statistical models with weakly stationary process assumptions have become standard in spatial statistics. However, one disadvantage of such models is the computation time, which rapidly increases with the number of data…
A growing body of work studies Blindspot Discovery Methods ("BDM"s): methods that use an image embedding to find semantically meaningful (i.e., united by a human-understandable concept) subsets of the data where an image classifier performs…
While advanced analysis of large dataset is in high demand, data sizes have surpassed capabilities of conventional software and hardware. Hadoop framework distributes large datasets over multiple commodity servers and performs parallel…
We have developed an end-to-end photometric data processing pipeline to compare current photometric algorithms commonly used on ground-based imaging data. This testbed is exceedingly adaptable, and enables us to perform many research and…
Huge amounts of data being generated continuously by digitally interconnected systems of humans, organizations and machines. Data comes in variety of formats including structured, unstructured and semi-structured, what makes it impossible…
Astronomy is increasingly encountering two fundamental truths: (1) The field is faced with the task of extracting useful information from extremely large, complex, and high dimensional datasets; (2) The techniques of astroinformatics and…
Hyperspectral cameras provide numerous advantages in terms of the utility of the data captured. They capture hundreds of data points per sample (pixel) instead of only the few of RGB or multispectral camera systems. Aerial systems sense…
MapReduce (and its open source implementation Hadoop) has become the de facto platform for processing large data sets. MapReduce offers a streamlined computational framework by interleaving sequential and parallel computation while hiding…
This project aims to build an elastic web application that can automatically scale out and scale in on-demand and cost-effectively by utilizing cloud resources, specifically from Amazon Web Services (AWS). The application is an image…
At present, and increasingly so in the future, much of the captured visual content will not be seen by humans. Instead, it will be used for automated machine vision analytics and may require occasional human viewing. Examples of such…
We propose a new, efficient multi-scale method to decompose a map (or signal in general) into components maps that contain structures of different sizes. In the widely-used wave transform, artifacts containing negative values arise around…