Fallen trees represent a vital component of forest ecosystems, and their accurate identification and segmentation is a crucial basis for forest resource inventory and dynamic monitoring. Due to the complex characteristics of fallen trees in images, such as variable morphology, significant scale differences and blurred boundaries, an instance segmentation algorithm named DSM-YOLO is proposed based on an improved YOLO11 framework. To address the irregular shapes of fallen trees, a dual dynamic task align head (DDTAH) is introduced to replace the original segmentation head. It jointly leverages multiple convolutional layers to extract features, improving inter-task complementarity and enhancing the network’s capacity for feature extraction and expression, thereby improving segmentation accuracy. To handle large-scale variations of fallen trees, a shared dilated convolution pyramid (SDCP) module is incorporated into the backbone network. It utilizes dilated convolution to capture multi-scale features, enabling the model to better adapt to objects of different sizes and ensuring segmentation accuracy across scales. To mitigate fuzzy edge-background transitions and inaccurate boundary localization, a multi-scale edge attention (MSEA) module is integrated into the backbone. It enhances edge-related features, making the model more sensitive to detailed contours of fallen trees, which reduces background interference and improves boundary segmentation. The experimental results indicate that the proposed model achieves a 4.3% improvement in mAP50 over the original YOLO11. It also outperforms other segmentation models in accuracy, demonstrating the effectiveness of DSM-YOLO and providing technical support for detailed forest resource surveys.