Abstract:The widespread adoption of encryption technology has given malicious activities a chance to hide, posing a great challenge to network security monitoring systems. Existing encrypted traffic detection methods primarily extract statistical traffic features at the individual packet level. However, this may disrupt the features implied in the original continuous communication behavior, due to potential IP fragmentation. Furthermore, most approaches model the interaction patterns between network flows at a relatively coarse granularity, failing to thoroughly explore the communication intent between peer entities. This study introduces a novel method, interaction state graph-net (ISG-Net), which uses interaction as the analysis granularity. ISG-Net constructs a traffic interaction state graph based on state transitions and applies a self-attentive encoder model to capture temporal traffic information. In particular, interaction state representations containing global information are obtained through the interaction state graph. Then, fine-grained features of each interaction are extracted to obtain the representation of the sessions (bidirectional flows). Experimental results on three datasets demonstrate that the proposed method outperforms existing methods in terms of accuracy, robustness and fault tolerance in the task of encrypted malicious traffic detection.