Abstract:Jailbreak attacks are crucial for identifying and mitigating security vulnerabilities in large language models (LLM). These attacks aim to bypass security mechanisms and induce models to produce prohibited outputs. However, it is difficult to directly and fairly compare these attacks, as they are typically evaluated on different data samples and models. This study introduces EasyJailbreak, a unified framework that simplifies the construction and evaluation of jailbreak attacks for LLMs and constructs jailbreak attacks by adopting four components, including the selector, mutator, constraint, and evaluator. This modular design allows researchers to easily combine existing and novel components to develop various attack methods. To demonstrate the utility of this framework, this study conducts extensive empirical evaluations, with 11 different jailbreak methods implemented based on this framework. Additionally, comprehensive security validations are performed on LLMs, involving over 750000 inference queries across 10 different LLMs. The results reveal an average breach probability of 60% under various jailbreak attacks. Notably, even advanced models like GPT-3.5-turbo and GPT-4 show average attack success rates of 57% and 33% respectively.