Active Metadata and Machine Learning based Framework for Enhancing Big Data Quality

Abstract

The advent of big data has ushered in a new era of opportunities across industries, facilitating transformative insights and operational enhancements. However, the inherent challenges of big data, including its voluminous nature, rapid generation pace, and heterogeneous sources, pose serious issues to data quality. Inaccuracies, incompleteness, and inconsistency undermine the integrity and reliability of analytics, necessitating robust solutions for quality assurance. Despite the recognition of these challenges, existing solutions often lack comprehensive and adaptable mechanisms to ensure data quality throughout its lifecycle. To address this gap, this paper proposes a novel framework using active metadata, enriched with machine learning capabilities, to enhance big data quality effectively and intelligently. The framework comprises five key steps, starting with metadata acquisition to provide foundational insights into data characteristics. Subsequent phases involve advanced preprocessing techniques, machine learning-based anomaly detection using acquired metadata, and correction of anomalies using predictive models within appropriate metadata neighborhoods. Based on active metadata and machine learning, the framework automatically identifies and rectifies discrepancies, thereby improving overall data reliability and usability. Experimental validation of the proposed framework using a large dataset demonstrates its efficacy in correcting quality anomalies.