Gray Swan ads

Prompt injections, jailbreaks, and reward hacking keeping you up at night? If your AI agents are tool calling, accessing internal data, sending messages, and making changes on your behalf, we'd like to know: How are you identifying the risks and enforcing guardrails around your agents?

Most safety and security teams don’t know what they don’t know until it’s too late – something I’ve consistently found in my meetings with them. This is often attributable to teams thinking that if they try a few different prompts and don’t see anything break, then they’re safe. A study we conducted with OpenAI, Anthropic, Meta, Google DeepMind, UK AI Security Institute (AISI), and National Institute of Standards and Technology (NIST) CAISI showed us how a false sense of security can leave companies wide open to indirect prompt injection attacks (study linked in the comments). Our data comes from a competition in the Arena with 464 participants; they submitted 272,000 attack attempts against 13 frontier models. The results: 8,648 successful attacks across 41 agent scenarios. And all models proved vulnerable. That’s roughly 1 in 30 attempts succeeding – far too high a number when sensitive customer data is at stake. The risk is particularly acute when it comes to open-weights models. Since the study, many of the closed models have gotten a lot more robust. That hasn’t been the case with the most vulnerable open-weights models. For companies testing their open-weights models with a series of prompts, it’s worth asking: what is your team missing?

AI security isn't cybersecurity. New attack surface, new failure modes, and a new playbook. Most enterprise security stacks weren't built for any of it. This AI Security 101 eBook is the toolbox for leaders securing models and agents in production.

Before a new model ships, Gray Swan's Arena is busy breaking it. Your enterprise AI systems deserve the same treatment.

Vendors offering AI security tools test against static lists only. In the real world, attackers don't use one. Battle-tested defenses start with adversaries that adapt.

So many AI vendors claim their system is secure. Gray Swan shows up in other people's system cards.