A Web Scraping Framework for Descriptive Analysis of Meteorological Big Data for Decision-Making Purposes

Abstract

With the large increase in the amount of heterogeneous, complex, and unstructured data issued from Web sources, there is an emerging need to develop Big Data technologies and tools to extract and manage this data. In this context, Web scraping for Big Data is a technique that has gained importance because of its rapidity and efficiency in gathering data for Big Data technology. This study was conducted to propose a Web Scraping framework for descriptive analysis of meteorological Big Data. The introduced framework makes it possible to extract a set of data to process it, present it in a form that is easier to analyze and understand, and use it for decision-making purposes about weather forecasts. The study employed descriptive analysis of available big data as it allows one to easily make quality and effective decisions. The web scraping process takes place in several stages, including data extraction, data archiving in a data warehouse, and finally data filtering and analysis. To test its applicability, the proposed web scraping framework was implemented and tested in the meteorological context to extract and present meteorological data issued from a specialized web source. The proposed system makes it possible to restore the data in the form of statistical models published in a dashboard. The results of the study revealed that the predictive models provided by the system are capable of predicting certain weather-related variables, such as humidity, precipitation, and temperature. The opportunities and implications to leverage the results of this study are many, including weather forecasting and decision support.