Pyramid Vision Transformer-Based Multi-Scale Feature Fusion and CNN Decoder for Accurate Colon Polyp Segmentation

Abstract

Polyp segmentation is essential in reducing the incidence of colorectal cancer by enabling the timely detection and removal of polyps, which are the precursors to most colorectal cancers. Accurate segmentation provides crucial information that aids in the early diagnosis and treatment of the disease. However, this task is challenging due to the considerable variation in polyp texture, color, and size. Additionally, some polyps closely resemble the surrounding healthy tissue in color, making it difficult to distinguish the boundaries between the polyp and its surrounding area. To address these challenges, we propose a deep learning model based on an encoder-decoder architecture. In the encoder section, we utilize a Pyramid Vision Transformer (PVT) to extract robust global feature representations across multiple stages. We introduce a multi-scale fusion module that combines features from different scales, allowing the model to capture diverse information. A custom convolution refinement attention module is employed to further refine the features, and it is integrated with the output of an efficient simple residual path, which preserves essential polyp features. Experiments are conducted on five publicly available polyp datasets. CVC-Clinic and Kvasir-SEG datasets are used for training, while CVC-ColonDB, Etis LaribDB, and CVC-300 datasets are used to evaluate generalization. The proposed approach achieves state-of-the-art performance, with a mean Dice coefficient of 93.9% and a mean Intersection over Union of 89% on CVC-Clinic, and 91.0% and 85.9% on Kvasir-SEG, respectively. Furthermore, our approach demonstrates the ability to adapt to new, unseen polyp features, achieving optimal results on the CVC-ColonDB, Etis LaribDB, and CVC-300 datasets. This adaptability makes it suitable for real clinical scenarios.