Abnormal driving behaviors of drivers pose a high risk of traffic accidents, threatening the life safety of drivers, passengers, and others. Thus, detecting the abnormal driving behaviors of drivers is of great significance for ensuring people’s travel safety. In an actual driving process, abnormal behaviors in the driver’s mouth region are complex and diverse. In view of this, this study proposes an unsupervised detection algorithm for the abnormal behaviors in the mouth region. The algorithm first uses the facial landmark detection network to obtain the mouth area with a high probability of anomaly. Then, the mouth area image is rebuilt with the improved Convolutional Auto-Encoder (CAE) algorithm. Abnormal behaviors are determined by the computation of the reconstruction error. The proposed algorithm is improved in three aspects: (1) the introduction of the skip connection structure to better reconstruct the input image; (2) the introduction of the Inception structure and the optimization of the proportions of branch channels to better fit the features of the input image; (3) the addition of Gaussian white noise in the training process to improve the robustness of model detection. The experimental results show that the AUC of the proposed algorithm framework is improved from 0.682 to 0.938 as compared with the traditional CAE algorithm and it can run on embedded systems.