A robot control system based on the hand gesture recognition is researched in this paper. First of all, this system uses image recognition technology and YCrCb skin color model to find the palm. Then it analyzes the fingertip and palm center. Furthermore, using frame difference method to identify the movement tendency of the hand and some simple gesture information. Finally, the signals recognized by the system will be transmitted to the robot. This is accomplished through the wireless Bluetooth serial port, in order to control the robot by the hand gesture. This system, which is compiled by OpenCV computer vision library under VS2010, can control the robot through some simple gesture and get rid of the physical contact during the Human-Machine interaction. The experimental result shows that this system can control the robot move forward, move backward, turn left, turn right, stop, speed up in real-time and the recognition rate is above 90%. This technology also has a high reference value on researches such as machine learning and automatic recognition.