In the Sloan digital sky survey (SDSS), the current object detection algorithm is inefficient in the detection of small-scale astronomical objects due to interference from large and bright astronomical objects. To address this issue, a small-scale astronomical object detection method based on Mask-GAN and improved YOLOv3 is proposed. The method is executed in two steps. The first step is to mask the interfering astronomical objects. A Mask construction algorithm for interfering astronomical objects is designed, which extracts the interfering objects by adaptive threshold segmentation and connectivity domain analysis, and the Mask is constructed by the method of fusing the features of band regions to avoid halo residue and excluding adjacent objects to avoid segmentation errors. Then, a GAN model is built, which is combined with the Mask of interfering astronomical objects to complete the interference masking task. The second step is to input the processed data into the improved YOLOv3 model for small-scale astronomical object detection. C-EfficientNet with an attention mechanism is built as the backbone network of the improved YOLOv3 to strengthen the feature extraction capability and increase the network’s attention to objects. Meanwhile, four effective feature layers are extended, and the method SAt is proposed to increase the weight of shallow feature maps so that the network can better use high-resolution shallow features with more details to detect small-scale astronomical objects. Experiments and analysis show that the average accuracy of the method in detecting small-scale stars and galaxies on the SDSS astronomical dataset reaches 81.16% and 77.89%, respectively, The proposed detection method is better than the classic one and is of certain practical application significance.