Existing multi-spectral pedestrian detection algorithms suffer from insufficient multimodal interaction and lack of long-distance dependence of fusion algorithms, thus resulting in poor performance in small-scale pedestrian detection in low-light conditions. To this end, this study proposes an fusion adaptive dual attention and axial attention Transformer network (ADATNet) for multi-spectral small-scale pedestrian detection. The method adopts the dual-branch CSPDarknet53 network to separately extract deep features from visible and infrared images, preserving the unique information from each modality. Meanwhile, two feature fusion modules are designed, including the adaptive dual attention module (ADAM) and axial attention Transformer feature enhancement module (ATFE). ADAM aims to enhance the model’s focus on critical features and suppress irrelevant or redundant information, while ATFE correlates the positional encoding of multimodal features to fuse the enhanced features, thereby both ensuring computational efficiency and capturing long-distance dependence. The fused features are then fed into the detection head to output the final detection results. Experimental results indicate that MR^–2 of ADATNet decreases to 7.08% on the KAIST dataset, while the mAP50 for the FLIR and LLVIP datasets reaches 82.8% and 97.6% respectively. They have an improvement of 4.7% and 1.9% over the baseline methods, demonstrating excellent detection performance.