agentaudit-eval is an open-source evaluation framework for multi-agent AI workflows. Below are 17 llm eval & observability apps with similar functionality to agentaudit-eval, matched by what each product actually does — not ranked or scored. Explore each to find the closest fit for your use case.
agentaudit-scanner is an open-source CLI tool that scans AI agent skills, rule files, and MCP configurations for security risks like prompt injections and hidden instructions. It is designed for AI security engineers and developers to ensure safe agent deployments.
openagent-eval is an open-source command-line framework designed for evaluating Retrieval-Augmented Generation (RAG) systems and AI agents. It provides tools and metrics for assessing LLM-based workflows, making it useful for AI researchers and developers who need to benchmark and analyze agent performance.
agentsec-eval is an open-source CLI framework for evaluating the security of AI agents. It provides adversarial test runners, server-side audits, and scoring mechanisms to help researchers and developers identify vulnerabilities such as prompt injection and improve agent robustness.
Agent evaluation toolkit
AI. NET ecosystem. The platform provides features such as tool usage validation, which allows users to assert on tool chains and verify that specific tools are called in the correct order with appropriate arguments. Stochastic evaluation is supported, enabling repeated runs of agent tasks to assess actual success rates and standard deviations, reflecting the non-deterministic nature of large language models. Workflow evaluation capabilities allow for the testing of multi-agent flows, including validation of executor order, edge traversal, and per-graph tool calls. Performance evaluation tools enable users to set and assert on service level agreements (SLAs) related to response times, total duration, and estimated costs. AgentEval includes model comparison functionality, letting users benchmark multiple models against defined metrics such as tool accuracy, relevance, and cost per request. The toolkit also supports recording and replaying agent interactions, which allows for consistent, repeatable evaluations without incurring additional API costs. Security evaluation is addressed through a Red Team module that tests agents against 258 attack probes across all 10 OWASP LLM Top 10 vulnerabilities, with MITRE ATLAS technique mapping. This module covers a wide range of attack types, including prompt injection, jailbreaks, PII leakage, and more, and supports both quick scans and advanced, customizable attack pipelines. Security compliance reports can be exported in PDF format. Memory evaluation is another key feature, with tools for benchmarking agent memory retention, recall depth, temporal reasoning, fact updates, cross-session persistence, and noise resistance. Results can be exported as interactive HTML reports. NET developers seeking to rigorously test, benchmark, and ensure the reliability, security, and performance of their AI agents before production use.
AgentEvals is an open-source tool designed to evaluate and score the behavior of AI agents using telemetry data captured from real production or test environments. By analyzing OpenTelemetry Protocol (OTLP) streams and Jaeger JSON traces, it enables users to assess agent performance and inference quality without the need to rerun or replay expensive large language model (LLM) calls. This approach allows for benchmarking agents before deployment and provides insights based on actual agent traces rather than synthetic replays. The platform offers several evaluation features, including the ability to define golden evaluation sets that describe expected agent behaviors, tool calls, and trajectories. AgentEvals supports flexible trajectory matching with strict, unordered, subset, or superset modes, enabling nuanced comparisons between expected and observed agent actions. Users can also create custom evaluators in Python, JavaScript, or any language of their choice and share them through a community registry. AgentEvals is accessible through both a command-line interface (CLI) and a web user interface (Web UI). The CLI is tailored for automation and integration into CI/CD pipelines, enabling teams to gate deployments based on agent behavior quality scores. The Web UI provides interactive capabilities for visually inspecting traces, browsing results, comparing runs, and drilling into detailed evaluations. Installation is available via Python wheel, and evaluations can be run directly against trace files. 0 license. Its focus on trace-driven evaluation and support for both automated and interactive workflows make it suitable for developers and teams seeking to ensure the reliability and quality of AI agent behavior before production deployment.
agent-audit-kit is an open-source CLI tool that scans MCP-connected AI agent pipelines for security vulnerabilities. It helps AI security engineers ensure the safety and compliance of agent-based workflows.
aehf is an open-source command-line framework for evaluating and benchmarking AI agents. It provides tools for standardized testing, comparison, and analysis of agent performance, making it useful for AI researchers and developers.
agent-diagnostics is an open-source framework for analyzing and annotating the behavior and reliability of coding agents. It provides tools for benchmarking, taxonomy creation, and structured evaluation, helping researchers and developers understand agent performance.
agentgrade is an open-source testing framework for multi-agent AI systems. It provides regression tests, credit assignment, and prompt patching, enabling developers and researchers to evaluate and improve agent workflows efficiently.
agent-skill-eval is an open-source CLI framework for evaluating the skills of code-generating agents across models like OpenCode, Claude Code, and Codex. It enables researchers and developers to benchmark agent performance using standardized tests.
agentassay is an open-source Python package designed for formal regression testing of non-deterministic AI agent workflows. It provides tools for verifying the reliability and correctness of agent-based systems, supporting stochastic and CI/CD testing scenarios. Ideal for AI developers and QA engineers working with complex agent architectures.
agent-genesis is an open-source SDK and CLI/API toolkit for evaluating and testing AI agents. It provides developers with tools to benchmark, analyze, and improve agent performance during the development lifecycle.
agent-audit-logger is an open-source Python middleware that wraps agent frameworks to log every tool call, providing SOC 2 audit trails for small teams and developers needing compliance and traceability.
Agents' Last Exam is a large-scale benchmark designed to evaluate AI agents on real-world, economically valuable professional workflows. The platform focuses on assessing agent performance on long-horizon tasks with verifiable outcomes, aiming to provide objective and comparable scores across a broad range of domains. It is led by Berkeley RDI in collaboration with over 300 industry experts and spans 55 sub-industries that encompass most major fields of professional work conducted on computers. The benchmark includes over 1,500 tasks, with a target of 5,000, covering areas such as animation and visual effects production in Adobe After Effects, 3D modeling in Siemens NX, game development tasks in Unreal Engine, simulation and mold flow analysis in Moldex3D, architectural modeling and energy analysis in Rhino 3D, and neuroimaging analysis in FSLeyes. These tasks are designed to reflect real professional challenges, ensuring that agent evaluations are grounded in practical, economically relevant scenarios. Agents' Last Exam provides features such as a leaderboard for tracking agent performance, access to detailed agent traces, and documentation to support contributors. The platform encourages community involvement, allowing users to contribute new tasks and participate in co-authorship. It is supported by an advisory committee comprising professors and researchers from leading academic institutions and is connected with contributors and partners from both academia and industry. evidence_sufficient": true}
agent-probe-ai is an open-source CLI tool for adversarial resilience testing of AI agents. It evaluates agent robustness against attacks like memory poisoning and tool misuse, helping AI researchers and security engineers identify vulnerabilities.
aeo-audit is an open-source command-line tool that scans websites to evaluate their Agent/Engine Optimization (AEO) readiness. It provides actionable scores and reports for SEO professionals and webmasters to improve site visibility for AI agents.
