AgentEval

AI. NET ecosystem.

The platform provides features such as tool usage validation, which allows users to assert on tool chains and verify that specific tools are called in the correct order with appropriate arguments. Stochastic evaluation is supported, enabling repeated runs of agent tasks to assess actual success rates and standard deviations, reflecting the non-deterministic nature of large language models. Workflow evaluation capabilities allow for the testing of multi-agent flows, including validation of executor order, edge traversal, and per-graph tool calls. Performance evaluation tools enable users to set and assert on service level agreements (SLAs) related to response times, total duration, and estimated costs.

AgentEval includes model comparison functionality, letting users benchmark multiple models against defined metrics such as tool accuracy, relevance, and cost per request. The toolkit also supports recording and replaying agent interactions, which allows for consistent, repeatable evaluations without incurring additional API costs. Security evaluation is addressed through a Red Team module that tests agents against 258 attack probes across all 10 OWASP LLM Top 10 vulnerabilities, with MITRE ATLAS technique mapping. This module covers a wide range of attack types, including prompt injection, jailbreaks, PII leakage, and more, and supports both quick scans and advanced, customizable attack pipelines. Security compliance reports can be exported in PDF format.

Memory evaluation is another key feature, with tools for benchmarking agent memory retention, recall depth, temporal reasoning, fact updates, cross-session persistence, and noise resistance. Results can be exported as interactive HTML reports. NET developers seeking to rigorously test, benchmark, and ensure the reliability, security, and performance of their AI agents before production use.