In multimodal sentiment analysis (MSA), different modalities have different contributions to the final sentiment judgment, and the dominant source of sentiment judgment varies dynamically depending on the input content. The existing methods typically fixate text as the dominant modality, but overlook the fact that in practical scenarios, sentiment polarity varies with tasks and environments, resulting in incomplete consistency with the sentiment of text modalities. Therefore, fixating on text as the dominant modality for sentiment analysis can easily lead to the omission of key sentiment clues. To address the problem of dynamic dominant modality selection, this study proposes a dynamic fusion framework of sentiment factors guided dominant modality selection (SFGDMS), which dynamically selects the dominant modality by calculating the importance of each modality via dual-perspective evaluation. Firstly, cross-attention mechanisms are adopted to capture inter-modal interactions, and a sentiment influence weight calculation formula is designed to quantify each modality’s global influence contribution from the positive perspective. Secondly, a counterfactual causal validation framework is introduced to quantify the necessity contribution strength of each modality from the negative perspective by removing target modalities and observing the prediction shifts. Subsequently, a dual-path sentiment information filtering-enhancement module is designed, which utilizes the dominant modality to filter sentiment-irrelevant and conflicting information from auxiliary modalities, and enhances the dominant modality by employing the filtered auxiliary modality features. Finally, a multimodal Transformer is employed to fuse the dominant modality features and auxiliary modality features. Experimental results on two publicly available datasets CMU-MOSI and CMU-MOSEI show that this model outperforms a series of baseline models.