ExploitBench

ExploitBench is a benchmarking platform designed to evaluate the exploitation capabilities of AI agents on the V8 JavaScript and WebAssembly engine, which is used in environments such as Chrome, Edge, Node.js, and Cloudflare Workers. Unlike traditional benchmarks that provide a binary assessment of whether a bug can be triggered, ExploitBench introduces a multi-tiered evaluation framework that measures an agent’s progression through the stages of exploitation, from simply reaching vulnerable code to achieving arbitrary code execution.

The platform structures its assessment using an "exploitation ladder" composed of five tiers: coverage (reaching patched code or functions), reproduction (causing a crash or demonstrating a bug), target primitives (such as building reusable exploit components inside the V8 sandbox like addrof, fakeobj, caged_read, and caged_write), generic primitives (arbitrary read/write and information leaks), and full control (arbitrary code execution or control-flow hijack). Sixteen specific capabilities are distributed across these tiers, and each is graded independently. Scoring is performed mechanically by a deterministic verifier built into V8’s standalone shell (d8), ensuring that no human intervention or LLM-based judgment is involved in the evaluation process.

ExploitBench presents benchmark results for various AI models, showing which capabilities each model has achieved on at least one V8 vulnerability. The platform allows for comparisons across different run parameters, such as seeds, turn budgets, and harness configurations. It features a leaderboard sorted by capability coverage, enabling users to see how models perform across the exploitation ladder. The "AutoNudge" feature automatically prompts stalled or quitting models to continue their progress, facilitating autonomous evaluation without human oversight.

For experimentation, users can follow provided instructions to run sample evaluations using Docker images and a server setup, allowing them to observe AI agent performance on specific vulnerabilities. ExploitBench is attributed to Seunghyun Lee and Prof. David Brumley of Carnegie Mellon University.