At present, Chinese speech recognition has made some achievements. However, due to regional differences in China, different place has different dialect, the Chinese recognition system has low recognition rate and poor performance in the dialect recognition. In order to solve the shortcomings of speech recognition system in dialect recognition, an isolated word recognition system of Hengyang dialect based on HTK is proposed. This method constructs the Hidden Markov Models (HMM), using phoneme as the basic recognition unit and using the HTK3.4.1 toolbox to extract the speech feature parameters of 39-dimensional Mel frequency cepstral coefficients (MFCC). Viterbi algorithm is used to train and match the model to achieve the isolated word speech recognition system of Hengyang dialect. The system's performances are compared under the different phoneme models and different Gaussian mixture numbers. The experimental results show that the system performance can be greatly improved by combining the 39-dimensional MFCC with 5 Gauss mixed numbers and HMM model.