Depression is a common mental disorder that severely affects patients’ daily lives. Recent research has made progress in automatically estimating depression severity from audio and visual modalities. However, relying on a single temporal scale makes it difficult to model dynamic changes in depression-related features over long-term sequences. In addition, depression-related features are non-uniformly distributed along the temporal dimension, and existing methods struggle to explicitly model changes in the temporal importance of depression-related features across the two modalities. To address these issues, a multi-scale temporal modeling and attention fusion method is proposed. First, a multi-scale temporal feature learning module is designed to learn depression-related multi-scale temporal features from long temporal sequences. To learn short-term depression-related changes while addressing the limited long-term modeling capability of dilated convolution, a multi-scale dilated convolution-Mamba module is proposed to model long-term depression trends and preserve multi-scale temporal receptive fields. Second, a bimodal temporal attention fusion module is designed to learn changes in the relative temporal importance of depression-related features across the two modalities. The Mamba model is used to model the joint changes of the two modalities and adaptively learn the temporal importance weights of depression-related features in the two modalities, thus dynamically weighting the more discriminative depression-related features at different time steps. Finally, experimental results on the AVEC2017 and AVEC2019 depression datasets show that the proposed method achieves MAE and RMSE values of 4.45/5.54 and 4.76/5.99, respectively, outperforming most classical networks, which validates its effectiveness in the bimodal depression severity estimation task.