Skip to content
Alternatives
Software like agent-eval
What else does this job. Matched on what each project does, not on who links to whom.
Closest first
- AgentEvalagenteval.devAI. NET ecosystem. The platform provides features such as tool usage validation, which allows users to assert on tool chains and verify that specific tools are called in the correct order with appropriate arguments. Stochastic evaluation is supported, enabling repeated runs of agent tasks to assess actual success rates and standard deviations, reflecting the non-deterministic nature of large language models. Workflow evaluation capabilities allow for the testing of multi-agent flows, including validation of executor order, edge traversal, and per-graph tool calls. Performance evaluation tools enable users to set and assert on service level agreements (SLAs) related to response times, total duration, and estimated costs. AgentEval includes model comparison functionality, letting users benchmark multiple models against defined metrics such as tool accuracy, relevance, and cost per request. The toolkit also supports recording and replaying agent interactions, which allows for consistent, repeatable evaluations without incurring additional API costs. Security evaluation is addressed through a Red Team module that tests agents against 258 attack probes across all 10 OWASP LLM Top 10 vulnerabilities, with MITRE ATLAS technique mapping. This module covers a wide range of attack types, including prompt injection, jailbreaks, PII leakage, and more, and supports both quick scans and advanced, customizable attack pipelines. Security compliance reports can be exported in PDF format. Memory evaluation is another key feature, with tools for benchmarking agent memory retention, recall depth, temporal reasoning, fact updates, cross-session persistence, and noise resistance. Results can be exported as interactive HTML reports. NET developers seeking to rigorously test, benchmark, and ensure the reliability, security, and performance of their AI agents before production use.
- ai-eval-forgegithub.comZero-dependency eval harness for LLM and agent regression testing. Scores outputs with exact, contains, regex, JSON, citation, and token-F1 checks. Compares two runs to flag regressions.
- evalitepypi.orgevalite is a lightweight, model-agnostic framework for evaluating AI agents. It provides developers with tools for defining evaluation tests and measuring agent behavior across models.
- playagentgithub.comAgent testing SDK. Instrument, assert, classify.
- openagent-evalpypi.orgopenagent-eval is an open-source command-line framework designed for evaluating Retrieval-Augmented Generation (RAG) systems and AI agents. It provides tools and metrics for assessing LLM-based workflows, making it useful for AI researchers and developers who need to benchmark and analyze agent performance.
- marketplace-evalgithub.comFramework for simulating heterogeneous agent pipelines as competing agents in an AI marketplace
- agent-interrogatorpypi.orgAn AI agent interrogation framework for identifying attack surface.
- agentaudit-evalpypi.orgagentaudit-eval is an open-source evaluation framework for multi-agent AI workflows. It provides tools for handoff quality scoring, failure attribution, loop detection, and cost guardrails, helping developers monitor and improve the reliability and efficiency of complex AI agent systems.
- agentbeacongithub.comMulti-agent orchestrator for AI coding tools
- agent-skillspypi.org<!-- ~ Copyright (c) 2025-2026 Datalayer, Inc. ~ ~ BSD 3-Clause License --> [](https://datalayer.io) [](https://github.com/sponsors/datalayer) # 🧰 Agent Skills [](https://pypi.org/project/agent-skills) **Reusable Agent Skills**: Create, manage, and execute reusable code-based tool compositions for AI agents. For more information, see the [Agent Skills community website](https://agentskills.io), the [specification](https://agentskills.io/specification), and the [integration guide](https://agentskills.io/integrate-skills). ## Overview Agent Skills provides a simple and powerful way for AI agents to build their own toolbox. Skills are Python files that compose MCP tools and other skills to accomplish specific tasks. Agent Codemode consumes skills from this package. If you are using agent-codemode, import skill utilities from `agent_skills`. ### How It Works 1. **Skills are code files**: Python files in a `skills/` directory with asyn
- agentsviewgithub.comLocal web viewer for AI agent sessions
- agent-examreadthedocs.ioagent-exam is an Apache-2.0 evaluation framework for testing agent skills across Claude Code, Codex CLI, Copilot CLI, and OpenCode. It is intended for developers building and benchmarking agent workflows.
- agent-probe-aipypi.orgagent-probe-ai is an open-source CLI tool for adversarial resilience testing of AI agents. It evaluates agent robustness against attacks like memory poisoning and tool misuse, helping AI researchers and security engineers identify vulnerabilities.
- agentlint-devpypi.orgLint, score, and optimize AI agent configuration files (CLAUDE.md, AGENTS.md, Cursor rules)
- agentanvilgithub.comContract-based testing framework for LLM agents — hybrid metrics (objective + LLM-as-judge + human), multi-agent and A2A protocol support, deterministic record/replay envelope.
- AgentEvalsaevals.aiAgentEvals is an open-source tool designed to evaluate and score the behavior of AI agents using telemetry data captured from real production or test environments. By analyzing OpenTelemetry Protocol (OTLP) streams and Jaeger JSON traces, it enables users to assess agent performance and inference quality without the need to rerun or replay expensive large language model (LLM) calls. This approach allows for benchmarking agents before deployment and provides insights based on actual agent traces rather than synthetic replays. The platform offers several evaluation features, including the ability to define golden evaluation sets that describe expected agent behaviors, tool calls, and trajectories. AgentEvals supports flexible trajectory matching with strict, unordered, subset, or superset modes, enabling nuanced comparisons between expected and observed agent actions. Users can also create custom evaluators in Python, JavaScript, or any language of their choice and share them through a community registry. AgentEvals is accessible through both a command-line interface (CLI) and a web user interface (Web UI). The CLI is tailored for automation and integration into CI/CD pipelines, enabling teams to gate deployments based on agent behavior quality scores. The Web UI provides interactive capabilities for visually inspecting traces, browsing results, comparing runs, and drilling into detailed evaluations. Installation is available via Python wheel, and evaluations can be run directly against trace files. 0 license. Its focus on trace-driven evaluation and support for both automated and interactive workflows make it suitable for developers and teams seeking to ensure the reliability and quality of AI agent behavior before production deployment.
- agent-beltgithub.comEvaluation harness for real headless CLI agents - reproducible multi-turn scenarios, rule + LLM scoring, cross-agent comparison
- agent-clipypi.orgA suite of AI-powered command-line tools for text correction, audio transcription, and voice assistance.
- agentbench-clipypi.orgagentbench-cli is an open-source command-line tool that allows AI developers and researchers to evaluate, test, and scan the behavior and safety of AI agents. It provides automated checks and reporting for agent performance and compliance.
- agent-docpypi.orgInteractive document sessions with AI agents
- agentcfggithub.comCLI tool for deploying and managing AI coding agent configurations (MCP servers, skills, instructions) across multiple providers.
- agent-genesisagent-genesis-ai.comagent-genesis is an open-source SDK and CLI/API toolkit for evaluating and testing AI agents. It provides developers with tools to benchmark, analyze, and improve agent performance during the development lifecycle.
- psi-agentpypi.orgA microkernel-style agent framework
- agentfluentgithub.comLocal-first agent analytics with prompt diagnostics
- ai-agent-rulesgithub.comManage user-level AI agent configurations
- agent-zoogithub.comSecurity harness for AI coding agents (Claude Code, Codex CLI, etc.) — mitmproxy payload inspection + TOML policy control.
- agent-skill-evalpypi.orgagent-skill-eval is an open-source CLI framework for evaluating the skills of code-generating agents across models like OpenCode, Claude Code, and Codex. It enables researchers and developers to benchmark agent performance using standardized tests.
- agentscore-clipypi.orgLighthouse for AI agent development environments
- agentype-cligithub.comLocal AI-agent usage analytics and persona archetypes
- tea-agentpypi.orgA self-evolving AI agent with dynamic toolkit management.
- detect_agentgithub.comDetect if code is running in an AI agent or automated development environment
- Agent Brain CLIgithub.comAgent Brain CLI - Command-line interface for managing AI agent memory and knowledge retrieval
- uAgentsgithub.comLightweight framework for rapid agent-based development
- agents-smithgithub.comClone AI agent configurations into any project
- agentclipgithub.comTurn AI agent QA runs into shareable slideshows.
Ranked by how close each one sits to agent-eval in the index, not by popularity. Back to agent-eval →