Duplicate Detection with Efficient Language Models for Automatic Bibliographic Heterogeneous Data Integration
Databases
2015-04-29 v1
Abstract
We present a new method to detect duplicates used to merge different bibliographic record corpora with the help of lexical and social information. As we show, a trivial key is not available to delete useless documents. Merging heteregeneous document databases to get a maximum of information can be of interest. In our case we try to build a document corpus about the TOR molecule so as to extract relationships with other gene components from PubMed and WebOfScience document databases. Our approach makes key fingerprints based on n-grams. We made two documents gold standards using this corpus to make an evaluation. Comparison with other well-known methods in deduplication gives best scores of recall (95\%) and precision (100\%).
Cite
@article{arxiv.1504.07597,
title = {Duplicate Detection with Efficient Language Models for Automatic Bibliographic Heterogeneous Data Integration},
author = {Nicolas Turenne},
journal= {arXiv preprint arXiv:1504.07597},
year = {2015}
}