It is like a fire department practicing with controlled burns. They don’t just read the manual on how fire behaves; they light a pile of brush in a safe zone to see if the hoses actually reach the flames and if the water pressure holds. You can’t know if you can put out a house fire if you’ve only ever looked at a diagram of a hydrant.
Anthropic is trying to do the same for cybersecurity. Instead of relying on synthetic benchmarks—which are essentially the “diagrams of hydrants” in the AI world—they are looking at real-world incidents to see if their models can actually help a bad actor. In their recent post, Investigating three real-world incidents in our cybersecurity evaluations, they describe moving from “safety” tests (can I trick the AI into saying something mean?) to “capability” tests (can the AI actually help me write a functional exploit?).
The shift is an admission that the industry has been focusing on the wrong thing. For years, the goal was to stop the model from being “helpful” to a hacker. But that usually just meant adding a layer of refusal—a filter that says, “I cannot help you with that.” This is the AI equivalent of putting a “Do Not Enter” sign on a door that isn’t actually locked. If the model still possesses the underlying capability to write the code, the “safety” is just a facade.
(I suspect they’re just covering their backs before someone else proves it).
The real news here isn’t that they are testing for cybersecurity risks—everyone is doing that. The news is the admission that they need to mirror real-world incidents to find the gaps. They are essentially admitting that their previous evals were too clean, too academic, and too far removed from how a script kiddie or a state actor actually operates.
Here is the problem: evaluating a model after the capability already exists is a lagging indicator. By the time Anthropic builds a benchmark based on a “real-world incident,” that incident has already happened. The vulnerability is already known, and the exploit is already in the wild.
Why are we pretending that a benchmark created by the company that built the model is an impartial audit? It isn’t. It’s an internal check. While it’s better than doing nothing, it doesn’t solve the fundamental tension of LLMs: the more capable a model is at coding, the more capable it is at hacking. You cannot have a world-class Python expert that is simultaneously incapable of understanding how a buffer overflow works. The two skills are the same skill.
The compute cost of running these red-teaming loops is probably astronomical (and likely higher than the cost of the actual safety filters), but it still doesn’t move the needle on the actual risk. Once a model is released into the wild, the “safety” layer is just a puzzle for the user to solve. We’ve seen this with every single model release since GPT-4. Users find a way around the refusal, and suddenly the “capability” that Anthropic is so carefully measuring is available to anyone with a decent prompt.
It is a lagging indicator.
If we continue to rely on this cycle—incident happens, benchmark is created, model is “patched”—we are just documenting the decline of the security moat. The models are getting smarter faster than the evals are getting smarter. We are essentially trying to build a fence around a flood that has already entered the living room.
By Q4, we will see a public exploit that specifically bypasses these new capability benchmarks because the red-teaming process is simply too slow to keep up with the actual speed of exploitation. The only way to actually secure these systems is to accept that the capability is there and move the defense to the infrastructure level, rather than trying to lobotomize the model into submission.
