By using a vote of one-against-one Support Vector Machine advantages of high classification algorithm accuracy, an improved expression recognition method was proposed in order to modify the Extreme Learning Machine's disadvantage of bad stability and poor classification accuracy. The method combines one-against-one classification algorithm with Extreme Learning Machine, which are consist of a new algorithm-OAO-ELM. First, the algorithm uses the ELM process classification as weak classifier when training sample by one-against-one. Then, these weak classifiers are combined into the finally strong classification. Prediction the results of classification, by votes to the class. Gabor facial expressional features, since the high-dimensional Gabor features are redundant; The dimensional principal component analysis is used to select these features. Experimental results based on the JAFFE database show that it obtains higher accuracy and better stability.