Abstract:Medical image segmentation aims to partition medical images into meaningful regions, such as tissues, lesions, and specific anatomical structures. Different from natural image segmentation, this task faces challenges including low contrast, complex structures and limited training data, which restrict the performance of vision foundation models pretrained on natural images. Inspired by prompt tuning in large language models, an adaptive point prompt generation agent is proposed to enable vision foundation models to improve medical image segmentation performance with a small amount of annotated data. To address the complexity of target structures, UNet++ is utilized as the policy network of the proposed agent to capture multi-scale features in medical images. To mitigate low contrast between regions-of-interest (ROI) and background, this study designs a coordinate-label joint action space and constructs a foreground-sensitive reward function based on feedback from the vision foundation model. The interaction mechanism between the vision foundation model and the task-specific deep network enables effective generalization with limited annotation data. Experiments on spinal cord tumor and kidney tumor datasets demonstrate that the proposed method achieves approximately 2% improvements over state-of-the-art baseline models in terms of both Dice and IoU metrics.