Comparative Analysis and Optimisation of Machine Learning Models for Regression and Classification on Structured Tabular Datasets

Abstract

This research entails comparative analysis and optimisation of machine learning models for regression and classification tasks on structured tabular datasets. The primary target audience for this analysis comprises researchers and practitioners working with structured tabular data. Common fields include biostatistics, insurance, and financial risk modelling, where computational efficiency and robust predictive performance are essential. Four machine learning techniques (i.e., linear/logistic regression, support vector machines (SVMs), Extreme Gradient Boosting (XGBoost), and Multi-Layered Perceptrons (MLPs)) were applied across 72 datasets sourced from OpenML and Kaggle. The datasets systematically varied by observation size, dimensionality, noise levels, linearity, and class balance. Based on extensive empirical analysis (72 datasets ×4 models ×2 configurations =576 experiments), it is observed that, understanding the dataset characteristics is more critical than extensive hyperparameter tuning for optimal model performance. Also, linear models are robust across various settings, while non-linear models, like XGBoost and MLP, perform better in complex and noisy environments. In general, this study provides valuable insights for model selection and benchmarking in machine learning applications that involve structured tabular datasets.