In order to solve the overlap between frames and improve the self-adaptability of the speech signal, an improved speech recognition system based on hidden Markov model(HMM) and genetic algorithm neural network is proposed in this paper. The major improvement is the adoption of wavelet neural networks in the training of Mel frequency cepstral coefficients(MFCC). And by using HMM models time series of speech signal, the speech's score on the output probability of HMM is calculated. The results will be used as the input of genetic neural network, the information of the speech recognition and classification can then be obtained. The experimental results show that, the improved system has better noise robustness than the pure HMM and the performance of the speech recognition system is also improved