Adaptive quadtree-based segmentation of nucleus and cytoplasm in pap-smear images: a lightweight and interpretable approach for automated cytology

Abstract

Background Automated analysis of Pap-smear images plays an important role in cervical cancer screening, particularly in low-resource settings where manual cytology remains labour-intensive, subjective, and prone to inter-observer variability. On the other hand, accurate segmentation of the nucleus and cytoplasm is a fundamental step in computer-aided diagnosis systems because it enables quantitative morphometric analysis and computation of clinically important biomarkers such as the nucleus-to-cytoplasm ratio. However, robust cervical cell segmentation remains challenging due to staining variability, inhomogeneity, irregular morphology, weak cytoplasmic boundaries, and overlapping cellular structures. This study presents an adaptive quadtree-based segmentation framework for automated nucleus and cytoplasm delineation in Pap-smear images. Methods The proposed method employs hierarchical split–merge decomposition guided by a dynamic adaptive statistical homogeneity analysis using mean intensity, variance, and entropy measures. Preprocessing is performed using large-kernel median filtering for background normalisation, followed by local Otsu thresholding, adaptive region merging, overlap refinement, and morphological post-processing. The framework was evaluated on both the Herlev cervical cytology dataset and the ISBI 2015 cervical cytology segmentation challenge dataset containing overlapping and clustered cervical cells. Comparative benchmarking was additionally performed against the U-Net and Attention U-Net. Results On the Herlev dataset, the proposed framework achieved nucleus Dice coefficients exceeding 0.94 and Zijdenbos Similarity Index (ZSI) values greater than 0.9034 across all diagnostic classes, with competitive cytoplasm segmentation performance. On the ISBI 2015 dataset, the framework maintained acceptable segmentation performance under overlapping-cell conditions, achieving nucleus Dice and ZSI values of 0.912 ± 0.048 and 0.918 ± 0.044, respectively. Morphometric feature comparisons demonstrated strong agreement with ground-truth annotations and low average percentage errors for area and diameter measurements. Although deep learning models achieved superior performance under highly complex overlap conditions, the proposed framework remained competitive while requiring substantially lower computational resources and no iterative model training. Conclusion The proposed Adaptive Quadtree-Based Segmentation framework provides a lightweight, interpretable, and computationally efficient approach for automated cervical cytology segmentation. Its training-free design, transparent statistical decision rules, and reduced hardware requirements make it particularly suitable for deployment in resource-constrained and embedded cervical cancer screening systems. The framework provides a practical segmentation backbone for automated cytology analysis and downstream computer-aided diagnosis applications.