In the application of safety monitoring system of elevators, infrared sensor technology or traditional face detection algorithms involving Haar-like and HOG features are often used for the recognition of elevator passengers with poor effect though. With the development of deep learning in recent years, the face detection algorithm based on convolutional neural networks is more accurate than traditional face detection algorithms and has been applied in many fields. Moreover, the face detection algorithm based on multi-task cascaded convolutional neural networks is adopted to recognize elevator passengers in the safety monitoring system owing to its small model and fast operation. With the inception module introduced, the depth and width of networks at all levels are raised by the parallel operation of convolutional cores of different sizes for better extraction of network features; models are trained faster and network classification is enhanced through batch normalization. The experimental results show that the accuracy of the improved algorithm is 2% higher than that of the original one and can thus realize the highly accurate recognition of elevator passengers.