On a construction site, safety helmets can reduce head injuries, and safety helmets of different colors represent different identities. The contemporary method of detecting safety helmet wearing and identifying types of work by video surveillance is time-consuming, incomplete, and low in supervision efficiency. In response, this study proposes an improved method of safety helmet wearing detection and identity recognition based on the you only look once version 4 (YOLOv4). On the basis of the original YOLOv4, the K-means algorithm is used to cluster the size of the prior box again, and multi-scale prediction output is added. The experimental distance intersection over union–non-maximum suppression (DIoU–NMS) is used for NMS so that safety helmet wearing detection and identity recognition of workers can achieve high efficiency and comprehensiveness. The results show that the average detection accuracy among workers wearing red, blue, yellow, and white safety helmets and workers without safety helmets is 92.1%, which means the proposed method ensures the real-time monitoring of the safety helmet wearing of workers on the construction site.