AgentEvals

AgentEvals is an open-source tool designed to evaluate and score the behavior of AI agents using telemetry data captured from real production or test environments. By analyzing OpenTelemetry Protocol (OTLP) streams and Jaeger JSON traces, it enables users to assess agent performance and inference quality without the need to rerun or replay expensive large language model (LLM) calls. This approach allows for benchmarking agents before deployment and provides insights based on actual agent traces rather than synthetic replays.

The platform offers several evaluation features, including the ability to define golden evaluation sets that describe expected agent behaviors, tool calls, and trajectories. AgentEvals supports flexible trajectory matching with strict, unordered, subset, or superset modes, enabling nuanced comparisons between expected and observed agent actions. Users can also create custom evaluators in Python, JavaScript, or any language of their choice and share them through a community registry.

AgentEvals is accessible through both a command-line interface (CLI) and a web user interface (Web UI). The CLI is tailored for automation and integration into CI/CD pipelines, enabling teams to gate deployments based on agent behavior quality scores. The Web UI provides interactive capabilities for visually inspecting traces, browsing results, comparing runs, and drilling into detailed evaluations. Installation is available via Python wheel, and evaluations can be run directly against trace files.

0 license. Its focus on trace-driven evaluation and support for both automated and interactive workflows make it suitable for developers and teams seeking to ensure the reliability and quality of AI agent behavior before production deployment.