To address the problem of low accuracy and poor real-time performance of the facial attractiveness evaluation system, we propose a new facial attractiveness evaluation system based on deep learning. In this system, the HOG feature-based method and the FaceNet pre-training model are used for face detection and facial feature extraction respectively. Furthermore, a two-layer decision model based on the Softmax classification layer and ReLU regression layer is proposed, which is combined with the quantized values of local facial features to evaluate the facial attractiveness. Experimental results on the SCUT-FBP5500 dataset show that the accuracy of the system is 78.58%, and the average evaluation time of a single image is 2.98 seconds, which can meet the needs of practical applications.