Abstract:Graph convolutional network (GCN) have been extensively applied to skeleton-based action recognition due to their superior modeling of non-Euclidean data structures. Since graph convolution has local aggregation properties, it focuses on neighborhood joints and short-time windows. However, most human actions depend more on global joint synergy. Existing methods based on graph convolution generally ignore the spatial connection topology and higher-order group relations. To this end, this study proposes a novel framework that integrates higher-order spatial modeling with multi-scale temporal modeling, which is called the spatio-temporal hypergraph Transformer (ST-HGT) network. A skeletal hypergraph is adopted to construct high-order spatial structures, and skeletal distance deviation is combined to dynamically model the overall relative displacement of skeleton joints and distant skeletal features. Meanwhile, to enhance the complementarity between skeleton edges and joints, the study utilizes graph convolution and the dynamic self-attention mechanism to jointly model edge and joint features in multi-frame sequences. Additionally, a multi-scale temporal convolution module is introduced to extract the skeletal temporal features under different time windows. The proposed ST-HGT has been extensively experimented on the three challenging benchmark datasets of NTU-RGB+D, NTU-RGB+D 120, and Northwestern-UCLA. The experiments show that the proposed method reaches or exceeds the precision of state-of-the-art methods on all three benchmark datasets.