Data Cleaning publication categorizer

The linkage of records which refer to the same entity in separate data collections is a common requirement in public health and biomedical research. Traditionally, record linkage techniques have required that all the identifying data in which links are sought be revealed to at least one party, often a third party. This necessarily invades personal privacy and requires complete trust in the intentions of that party and their ability to maintain security and confidentiality.

An important prerequisite to successfully integrating protein data is detecting duplicate records spread across different databases.

As databases become more pervasive through the biological sciences, various data quality concerns are emerging. Biological databases tend to develop data quality issues regarding data legacy, data uniformity and data duplication. Due to the nature of this data, each of these problems is non-trivial and can cause many problems for the database. For biological data to be corrected and standardised, methods and frameworks must be developed to handle both structural and traditional data. This paper discusses issues concerning biological data quality with respect to data cleaning.

As databases become more pervasive through the biological sciences, various data quality issues regarding data legacy, data uniformity and data duplication arise. Due to the nature of this data, each of these problems is non-trivial. For biological data to be corrected and standardized, new methods and frameworks must be developed. This paper proposes one such framework, called BIO-AJAX, which uses principles from data cleaning to improve data quality in biological information systems, specifically in TreeBASE.

Similarity-based grouping of data entries in one or more data sources is a task underlying many different data management tasks, such as, structuring search results, removal of redundancy in databases and data integration. Similarity-based grouping of data entries is not a trivial task in the context of life science data sources as the stored data is complex, highly correlated and represented at different levels of granularity. The contribution of this paper is two-fold. 1) We propose a method for similarity-based grouping and 2) we show results from test cases.

Naturwissenschaftliche Daten sind aufgrund
ihres Entstehungsprozesses oft mit
einem hohen Maß an Unsicherheit behaftet.
Bei der Integration von Daten aus verschiedenen
Quellen führen diese Unsicherheiten,
neben der vielfältigen syntaktischen
und semantischen Heterogenität in
der Repräsentation von Daten, zu Konflikten,
die in einer verringerten Qualität des
integrierten Datenbestandes münden. Obwohl
Konflikte oftmals nur durch Domänenexperten
endgültig aufgelöst werden
können, kann und muss die Arbeit dieser
Experten durch geeignete Werkzeuge unterstützt

This manual describes prototype software called Febrl designed to undertake probabilistic data cleaning (or standardisation) and record linkage. Written in the Python programming language, this software aims to allow health, biomedical and other researchers to clean (standardise) and link data sets of all sizes faster, with less effort and with improved quality.