When Fernando Iturrieta deployed a legal-domain AI assistant and invited the public to break it, roughly 2,000 people accepted the challenge. Rather than treating this as a stunt, he treated it as a structured red-team exercise—logging every interaction to understand the real attack surface of a production AI tool. The findings are more instructive than most theoretical security writeups.
The most common attack category was straightforward prompt injection: users tried to override system instructions by inserting commands like "ignore previous instructions" or claiming elevated permissions. A significant share attempted role-playing gambits—asking the assistant to pretend it was a different, unconstrained AI. Both approaches are well-documented, but seeing their frequency in a real deployment underscores that they remain the default moves for most adversarial users.
More interesting were the social-engineering attempts: users constructed elaborate fictional scenarios designed to make harmful or out-of-scope responses seem contextually justified. These are harder to block with simple keyword filters because the problematic intent is embedded in plausible-sounding narrative. The assistant's domain-specific grounding—being explicitly scoped to legal questions—provided a natural containment layer that generic assistants lack.
On the defensive side, the experiment validated a few practical principles. Clear, specific system prompts that define scope positively (what the assistant does) outperformed prompts that tried to enumerate prohibitions. Layered output filtering caught edge cases that prompt-level defenses missed. No single mechanism was sufficient; the resilient responses came from overlapping controls.
The practical takeaway for builders: if you're shipping an AI assistant to real users, assume adversarial input from day one and instrument your logs to categorize it. A controlled public red-team—even an informal one—will surface attack patterns faster than internal testing. Domain restriction is a genuine security property, not just a product decision. And the gap between "it works in demos" and "it holds under hostile use" is exactly where production AI systems fail.