A system for linking and integrating heterogeneous datasets in health and demographic surveillance systems

Abstract

To overcome the constraints of data silos within data-driven organizations, the linking and integration of data from several sources becomes even more crucial. Data linking and integration not only improve the precision of the analysis, but also discover hidden trends, patterns, and relationships that can be overlooked when working with separate datasets. Taking this into account, this study is aimed at linking and integrating data across disparate data sources in the Health and Demographic Surveillance System (HDSS). Data linkage was achieved by implementing a fuzzy pattern-matching algorithm (Levenshtein distance) written in structured query language (SQL). A user-friendly graphical user interface (GUI) was developed using the C# programming language to facilitate the process of data matching and linking across various datasets. The Pentaho data integration (PDI) tool facilitated the integration of data from various data sources into a common database. These open-source technologies facilitated efficient linking and integration of data, enabling researchers to holistically discover patterns and trends in the data. The proposed algorithm achieved 98.1% precision and recall of 88.6% whilemaintaining 93.0% of good balance between avoiding false matches and missed matches.