For fine-grained marine organism classification in complex underwater environments, challenges such as high class similarity, strong background interference, and significant semantic ambiguity persist. To address these challenges, this study proposes a marine organism classification method that integrates hierarchical semantic modeling and cross-modal feature modulation. First, a hierarchical class pre-segmentation and discriminative routing module is constructed to reconstruct the fine-grained class space into a semantically consistent hierarchical class structure. High-confidence expert routing is achieved by combining CLIP semantic discrimination and GFNet structural discrimination. Second, a contextual attribute semantic reasoning module is designed to transform symbolic contextual attributes into textual descriptions that can be employed by CLIP, and stable semantic priors are obtained through conditional probability modeling. A structure-semantic consistency adjudication mechanism is further proposed to reduce the risk of routing errors. At the feature fusion level, a text modulation fusion control module and a semantic gating residual fusion mechanism are introduced to adaptively inject global semantic information into high-level and multi-scale visual features. Experimental results show that this method effectively improves classification stability and robustness in complex marine organism classification tasks.