openagent-eval is an open-source command-line framework designed for evaluating Retrieval-Augmented Generation (RAG) systems and AI agents. Below are 19 llm eval & observability apps with similar functionality to openagent-eval, matched by what each product actually does — not ranked or scored. Explore each to find the closest fit for your use case.
agentsec-eval is an open-source CLI framework for evaluating the security of AI agents. It provides adversarial test runners, server-side audits, and scoring mechanisms to help researchers and developers identify vulnerabilities such as prompt injection and improve agent robustness.
Agent evaluation toolkit
agentaudit-eval is an open-source evaluation framework for multi-agent AI workflows. It provides tools for handoff quality scoring, failure attribution, loop detection, and cost guardrails, helping developers monitor and improve the reliability and efficiency of complex AI agent systems.
AI. NET ecosystem. The platform provides features such as tool usage validation, which allows users to assert on tool chains and verify that specific tools are called in the correct order with appropriate arguments. Stochastic evaluation is supported, enabling repeated runs of agent tasks to assess actual success rates and standard deviations, reflecting the non-deterministic nature of large language models. Workflow evaluation capabilities allow for the testing of multi-agent flows, including validation of executor order, edge traversal, and per-graph tool calls. Performance evaluation tools enable users to set and assert on service level agreements (SLAs) related to response times, total duration, and estimated costs. AgentEval includes model comparison functionality, letting users benchmark multiple models against defined metrics such as tool accuracy, relevance, and cost per request. The toolkit also supports recording and replaying agent interactions, which allows for consistent, repeatable evaluations without incurring additional API costs. Security evaluation is addressed through a Red Team module that tests agents against 258 attack probes across all 10 OWASP LLM Top 10 vulnerabilities, with MITRE ATLAS technique mapping. This module covers a wide range of attack types, including prompt injection, jailbreaks, PII leakage, and more, and supports both quick scans and advanced, customizable attack pipelines. Security compliance reports can be exported in PDF format. Memory evaluation is another key feature, with tools for benchmarking agent memory retention, recall depth, temporal reasoning, fact updates, cross-session persistence, and noise resistance. Results can be exported as interactive HTML reports. NET developers seeking to rigorously test, benchmark, and ensure the reliability, security, and performance of their AI agents before production use.
agent-skill-eval is an open-source CLI framework for evaluating the skills of code-generating agents across models like OpenCode, Claude Code, and Codex. It enables researchers and developers to benchmark agent performance using standardized tests.
AgentEvals is an open-source tool designed to evaluate and score the behavior of AI agents using telemetry data captured from real production or test environments. By analyzing OpenTelemetry Protocol (OTLP) streams and Jaeger JSON traces, it enables users to assess agent performance and inference quality without the need to rerun or replay expensive large language model (LLM) calls. This approach allows for benchmarking agents before deployment and provides insights based on actual agent traces rather than synthetic replays. The platform offers several evaluation features, including the ability to define golden evaluation sets that describe expected agent behaviors, tool calls, and trajectories. AgentEvals supports flexible trajectory matching with strict, unordered, subset, or superset modes, enabling nuanced comparisons between expected and observed agent actions. Users can also create custom evaluators in Python, JavaScript, or any language of their choice and share them through a community registry. AgentEvals is accessible through both a command-line interface (CLI) and a web user interface (Web UI). The CLI is tailored for automation and integration into CI/CD pipelines, enabling teams to gate deployments based on agent behavior quality scores. The Web UI provides interactive capabilities for visually inspecting traces, browsing results, comparing runs, and drilling into detailed evaluations. Installation is available via Python wheel, and evaluations can be run directly against trace files. 0 license. Its focus on trace-driven evaluation and support for both automated and interactive workflows make it suitable for developers and teams seeking to ensure the reliability and quality of AI agent behavior before production deployment.
Benchmark your AI agent / RAG pipeline on the AI01 leaderboard.
tool-eval is a command-line framework for evaluating the tool usage of AI agents. It provides researchers and developers with tools to analyze and benchmark agent interactions with external tools.
openadapt-evals is an open-source infrastructure for evaluating and benchmarking GUI agents. It provides tools for automation, benchmarking, and performance analysis, supporting AI researchers and developers working on agent-based systems. The package is MIT licensed and available via PyPI and GitHub.
coder-eval is an open-source command-line tool for evaluating, benchmarking, and A/B testing AI coding agents. It uses sandboxed, reproducible YAML task suites, making it suitable for AI researchers and developers assessing agent performance.
Zero-dependency eval harness for LLM and agent regression testing. Scores outputs with exact, contains, regex, JSON, citation, and token-F1 checks. Compares two runs to flag regressions.
EvalSurfer is an open-source CLI tool that provides a skill-first, agent-native evaluation protocol for AI applications. It supports operational metrics and integrates with the Model Context Protocol (MCP), making it suitable for developers and researchers who need robust evaluation of LLMs and agentic systems.
evalgrid is an open-source framework designed for evaluating, scoring, and tracking the correctness of AI agents at scale. It provides tools for running large-scale agent tests, collecting metrics, and analyzing results, making it ideal for AI researchers and developers who need robust evaluation pipelines.
Evaluation module for Ragbits components
evalforge is an open-source CLI harness for evaluating LLM agents, designed to work across different frameworks. It helps AI researchers and developers assess agent performance and behavior.
leanlab is an open-source CLI tool designed for evolving and evaluating AI agent experiments and coding tasks. It supports experiment evolution against fixed metrics and automates the spec-to-merge workflow with locked acceptance tests, helping AI researchers and developers improve agent reliability and performance.
Self-hosted AI agents — streaming chat, tool use, persistent memory, multi-agent teams
gooddata-eval is an open-source CLI tool for evaluating the GoodData AI agent against user-defined questions and models. It enables data scientists and AI engineers to benchmark and compare AI agent performance in analytics workflows.
Evalgent provides a comprehensive platform for testing and evaluating AI voice agents, allowing teams to define real-world scenarios, configure caller personas, and measure custom metrics. It helps organizations catch behavioral failures, edge cases, and regressions before real users encounter them, improving the reliability of deployed voice agents. Designed for QA teams and AI product managers.
