Soybeans include many varieties (cultivars) and their cultivars have subtle differences in leaf patterns, which makes it tough to distinguish them from leaf features. Great progress has been made in using leaf image patterns for plant species recognition. However, as a general fine-grained pattern recognition problem, soybean cultivar recognition has not yet received considerable attention. Traditional hand-operated leaf image analysis is limited to capture the subtle differences of leaf features among different cultivars. In this study, we attempt to use deep learning to harvest discriminatory leaf features for soybean cultivar recognition. A novel deep learning model, Transformation Attention Network (TAN), is proposed in this work. It first extracts fine-grained leaf features via the attention mechanism and then rectifies the leaf posture by affine transformations. We construct a soybean leaf cultivar dataset which consists of 240 soybean cultivars, with 10 samples per cultivar, to examine the availability of cultivar information in leaf patterns and validate the effectiveness of the proposed deep learning model for soybean cultivar recognition. The experimental results confirm the effectiveness of the leaf image patterns in distinguishing cultivars and demonstrate the better performance of the proposed method than that of the state-of-the-art hand-operated methods and deep learning methods in soybean cultivar recognition.