Abstract:Given the problems of insufficient robustness, low efficiency of multimodal information fusion, and high computational complexity in existing audio-visual speech enhancement methods under complex scenarios, a dual-stream gated audio-visual fusion (DS-GAVF) architecture is proposed. This architecture achieves performance improvement in speech enhancement via static-dynamic visual collaborative modeling and fine-grained cross-modal interaction. In the feature extraction stage, U-Net is employed to encode the audio’s time-frequency characteristics, and meanwhile a dual-stream visual network is designed. ResNet-18 is adopted to extract static features of single-frame facial images, and a spatiotemporal graph convolutional network is utilized to capture dynamic motion patterns of continuous facial landmarks. Additionally, a dynamic temporal interpolation alignment method is proposed to solve the time series difference of audio-visual modalities, and a novel gated cross-attention mechanism is designed, which dynamically adjusts the feature fusion weights via a gated unit and suppresses noise interference when visual reliability is low. In the decoding stage, cross-modal jump connections are leveraged to inject multi-scale visual cues into the U-Net decoding layers, finally outputting the time-frequency mask of the target speech. Experimental results show that DS-GAVF yields excellent performance under mixed noise environments on three datasets. Compared with existing methods, DS-GAVF achieves a synergistic improvement in speech quality, intelligibility, and robustness while maintaining low computational complexity.