Abstract:The precise instance segmentation and simultaneous classification of nuclei in histopathological images present significant challenges, primarily due to high cellular density and morphological heterogeneity. To tackle the intertwined tasks of nucleus semantic segmentation and instance-level classification, we propose a novel multi-branch parallel decoder framework. Our approach enhances the canonical Hover-Net by replacing its convolutional backbone with a Swin Transformer architecture. To further improve its feature representation capability, we embed an Atrous Spatial Pyramid Pooling (ASPP) module within the Swin Transformer blocks, empowering the model to effectively extract and fuse multi-scale features across diverse receptive fields. We conducted a comprehensive evaluation of our framework on two challenging public datasets, CoNSeP and MoNuSeg. Despite a marginal increase in inference time, our model achieved state-of-the-art performance, attaining a Dice coefficient of 0.8561 and an IoU of 0.7630, which surpasses the baseline model and other leading methods in precise nucleus instance segmentation. Extensive ablation studies and qualitative visualizations further confirm the efficacy of the Swin Transformer and the ASPP module, highlighting the model's exceptional robustness in complex scenarios involving heavily overlapping nuclei and indistinct boundaries. This study introduces an effective solution for nuclear analysis in complex histopathology images, contributing to the development of more precise and reliable automated computational pathology diagnostics.