agentprdiff is a Python library and command-line interface designed for snapshot testing of large language model (LLM) agents. It addresses the challenge of testing stochastic systems, such as LLM agents, where traditional unit testing fails because outputs can vary between runs. By recording an agent’s behavior during a known-good run and committing this record to git, agentprdiff enables developers to compare subsequent runs, identifying changes in agent outputs, tool-call sequences, costs, and latencies. If any regression is detected, the CLI exits with a non-zero status, allowing CI builds to fail and preventing unnoticed behavioral drift from reaching production.
The tool is built to support fast, deterministic checks for the majority of agent behaviors that can be encoded as rules, such as verifying tool usage, keywords in output, or cost thresholds. For aspects that require semantic judgment, agentprdiff offers a pluggable grader system, supporting OpenAI, Anthropic, custom callables, or a deterministic fake judge for environments without API keys. It provides a suite model that uses simple Python files, avoiding the need for DSLs or YAML, and includes a set of built-in graders for common assertions. Baselines are stored as JSON files under a dedicated directory, and reviewers can see diffs directly in pull requests.
md file at the repository root to facilitate adoption. The CLI offers commands for initializing, recording, checking, reviewing, scaffolding, and diffing test suites. It supports adapters for OpenAI and Anthropic SDKs, and is compatible with providers like Groq, Gemini, OpenRouter, Ollama, vLLM, Together, Fireworks, and DeepInfra. Outputs include rich terminal tables and optional JSON artifacts, with CI-friendly behaviors such as exit codes on regression.
The platform is intended for developers and teams working with LLM agents who need to monitor and control behavioral changes during development and continuous integration. agentprdiff is not a framework and does not require changes to agent code, instead focusing on recording and asserting agent behavior across time.
agentprdiff is a LLM eval & observability product. It focuses on detecting behavioral regressions in LLM agents before code merges. agentprdiff is an open-source project aimed at AI/ML developers and teams using LLM agents. The project is open source (MIT). agentprdiff is available on the command line.
Behind agentprdiff is vnageshwaran-de, and the product first shipped in 2026. The project is developed in the open on GitHub with 13 stars and 44 commits in the last 90 days. Among its 5 catalogued features are snapshot testing, behavioral regression detection, and CLI integration.
Latest indexed changes and source events
Other apps tracked under the same category.
