Chinese AI developer Moonshot is conducting an internal review after researchers were able to persuade two of its popular Kimi models, Kimi K2.6 and K3 Swarm, to provide information on making biological weapons and carrying out assassinations. Mindgard, a company that tests the security of AI systems, reported to the BBC that it discovered in July that these models could evade safety limits established by their developers. This occurred during a process known as "jailbreaking," where researchers use complex instructions to test if AI tools ignore safety measures. Mindgard stated that these measures should have prevented Kimi from discussing sensitive topics.
Moonshot expressed to the BBC that it values third-party input as essential for improving AI safety and is in discussions with Mindgard regarding its findings. Mindgard's founder, Peter Garraghan, indicated that the ability of Kimi models to discuss any topic post-jailbreak is concerning, noting that they could provide creative and inventive recommendations on nefarious subjects.
The risks associated with jailbreaks differ from those seen in recent high-profile AI incidents involving autonomous AI tools developed by US firms like OpenAI, Meta, and Anthropic, which have been reported to hack online services. While jailbreaks are complex and time-consuming, experts worry that malicious actors could exploit them to cause harm. Anthropic recently announced it had disrupted attempts to use one of its AI models for malicious activities related to biological weapons development.
Mindgard has not confirmed whether the information provided by Kimi regarding sensitive topics would be effective, but it argues that guardrails should have prevented the models from engaging in such discussions. The firm also suggested that a jailbroken Kimi 2.6 could enable hackers to execute code on its computing resources and access the internet, potentially serving as a launchpad for cyber-attacks.
Garraghan defended Mindgard's decision to publicly discuss the jailbreak, stating that the company had informed Moonshot and did not disclose key details about how it achieved the jailbreak. Mindgard notified Moonshot of the jailbreak via email on July 27 and followed up about a week later, but Moonshot only contacted Mindgard recently after being approached by the BBC for comment.
In an email shared with the BBC by Moonshot, the company stated that its model generally showed a high refusal rate for such requests in internal evaluations. Moonshot's Kimi K3 model claims to rival those of OpenAI and Anthropic.
The findings emerge amid ongoing debates in the AI industry regarding the safety of closed, proprietary models versus open-source tools. Kimi is classified as an open-weight model, allowing individuals to run it on their own computing infrastructure. Professor Alan Woodward from the University of Surrey noted the risk of open-source models falling into the wrong hands, although they could also be utilized for cyber-defense. He mentioned that AI firm Hugging Face used a Chinese open-source model to analyze a hack attributed to OpenAI agents. Professor Woodward expressed skepticism about the pace of international regulation keeping up with AI development, stating, "It's taken us decades to agree on the format of telephone numbers." Like Garraghan, he believes there should be more focus on identifying and prosecuting individuals who misuse AI.