Tragedies caused by not wearing safety helmets occur from time to time in engineering sites. To assist site managers in protecting workers’ safety, this study has designed and implemented an intelligent safety helmet supervision system based on deep learning. The system adopts the YOLOv4 target detection model integrating speed and accuracy, generates the new anchor boxes by K-means clustering analysis on the data set, and then trains the model with the new anchor boxes. The detection accuracy of the helmet is improved to 92%. The detection model YOLOv4 is combined with the tracking model DeepSORT to effectively solve the problems of repeated warnings and failures to produce statistics on illegal data. Finally, it is built into a cross-platform mobile APP, which is convenient for managers to use the mobile terminal to monitor the helmet-wearing situation anytime and anywhere. This intelligent supervision system covers a set of functions including safety helmet detection, real-time broadcast of detection videos, intelligent warning, illegal picture capture and display, and illegal data visualization. It can greatly improve the production safety factors and supervision efficiency in the project site.