Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents

The non-deterministic nature of agents and the entanglement of traditional software vulnerabilities with LLM-specific risks make security analysis challenging. In this talk, we introduce threat snapshots: a framework that isolates execution states where LLM vulnerabilities propagate to the agent level. We will present the b3 benchmark, built from 194,331 crowdsourced attacks, and share results from evaluating 31 LLMs. Key finding: reasoning capabilities improve security, model size does not. We will discuss our framework, results, and what they mean for building secure LLM-based agents.

You can join the talk also online via this Zoom link: Meeting ID: 623 8535 9151 Passcode: 292957