In order to solve the problem of difficult recognition due to the large variety of handwritten calligraphy fonts and reduce the threshold for people to appreciate calligraphy, we propose a handwritten calligraphy font recognition algorithm based on deep learning. In the process of recognition, image processing methods, such as projection method, are first used to locate and segment the Chinese characters in the calligraphy works. Then, the GoogLeNet Inception-v3 model and ResNet-50 residual network are used to recognize the styles and shapes of Chinese characters. Consequently, this algorithm can recognize the styles and shapes of regular script and seal script in an entire calligraphy work at single-character recognition rates of 91.57% and 81.70%, demonstrating its practicability.