We describe a novel, "focusable", scalable, distributed web crawler based on GNU/Linux and PostgreSQL that we designed to be easily extendible and which we have released under a GNU public licence. We also report a first use case related to an analysis of Twitter's streams about the french 2012 presidential elections and the URL's it contains.
Cite
@article{arxiv.1212.5633,
title = {Design, implementation and experiment of a YeSQL Web Crawler},
author = {Pierre Joulin and Romain Deveaud and Eric SanJuan-Ibekwe and Jean-Marc Francony and Françoise Para},
journal= {arXiv preprint arXiv:1212.5633},
year = {2012}
}