The paper proposed a new algorithm used for hand gesture recognition which based on the convolutional neural network. The method not only avoids the hand gesture in the early period of the complex pretreatment, but also can directly input the gesture of original image. The convolutional neural network is characterized by local receptive field, hierarchical structure, global learning for feature extraction and classical. It has been applied to many image recognition tasks. Experimental results showed that the multi-class hand gestures can be recognized with high accuracy, small complexity and good robustness, while the inherent shortcomings of the traditional algorithm are overcame.