Accurate detection of pavement cracks is essential for traffic safety, yet existing models often struggle to capture the detailed features of fine and irregular cracks, limiting their engineering applicability. This study aims to enhance the precision and practicality of crack detection. A novel image segmentation method is proposed by integrating domain knowledge-structured CLIP with the FastSAM image segmentation model. First, a structural knowledge graph of asphalt pavement crack distress is constructed, and a dedicated knowledge structuring encoder is designed. Subsequently, a two-stage fusion strategy is implemented. The CLIP model is fine-tuned with the structured knowledge to strengthen its semantic representation of cracks. The enhanced CLIP features are then embedded into the FastSAM encoder to guide the segmentation process. Experimental results show that the knowledge structured CLIP outperforms existing methods on both the custom-built CBAPDD-30K and the public MSCOCO datasets, significantly improving the accuracy of fine- grained semantic image matching. On the CBAPDD-30K dataset, the proposed two-stage fusion model achieves superior performance. In server-side experiments, it attains a precision of 91.67%, a recall of 90.87%, an F1-score of 91.27%, and an mIoU of 81.91%, surpassing mainstream lightweight and multi-modal models. When deployed on edge devices, the model exhibits efficient real-time performance in terms of FPS and inference time. This method achieves a favorable balance between detection precision and speed. It possesses high engineering application value.