Fine-Tuning Pretrained Language Models for Automated Research Papers Classification

Abstract

The exponential growth of academic publications across diverse disciplines presents a major challenge for effective indexing, retrieval, and recommendation. In this study, we explore the use of large language models (LLMs) for the automatic classification of research papers into academic fields based solely on their abstracts. Specifically, we investigate the effectiveness of fine-tuning GPT-2 model variants Small, Medium, Large, and XL for this task. Our goal is to use the semantic understanding capabilities of LLMs to overcome the limitations of traditional keyword-based classification methods. We evaluate two fine-tuning strategies: (i) adapting only the classification head while freezing the rest of the model, and (ii) fine-tuning the last transformer block along with the classification head. Experimental results, measured by Accuracy, F1-Score, and ROC-AUC, reveal that deeper fine-tuning consistently yields superior performance. Among the evaluated models, GPT-2 XL achieved the highest classification accuracy. This work underscores the promise of domain-adapted LLMs in enhancing scholarly content organization and lays the groundwork for intelligent academic recommendation and retrieval systems.