Deep learning has become a popular research direction recently and has been applied to many fields. But the performance issue caused by the complicated learning model and massive training data has becomes an obstacle of deep learning evolution. As the development of processor technology, the core number and performance of the processor have increased rapidly. However, the acceleration research is restricted mainly due to the low parallelism and high memory consumption of the training algorithm. The paper introduces the background of deep learning and its training algorithm, then summarizes current deep learning acceleration works. Further, it analyzes classic deep models training algorithm to explain the causes of deep learning performance problem. Based on the analysis, it lists the challenges about deep learning evolution and makes proposal to address them.