coder-eval is an open-source command-line tool for evaluating, benchmarking, and A/B testing AI coding agents. Below are 10 llm eval & observability apps with similar functionality to coder-eval, matched by what each product actually does — not ranked or scored. Explore each to find the closest fit for your use case.
caliper-eval is an open-source CLI tool designed for evaluating Claude Code skills and AI agents. It provides developers and researchers with tools to assess and benchmark AI agent performance from the command line.
openagent-eval is an open-source command-line framework designed for evaluating Retrieval-Augmented Generation (RAG) systems and AI agents. It provides tools and metrics for assessing LLM-based workflows, making it useful for AI researchers and developers who need to benchmark and analyze agent performance.
Benchmark your AI agent / RAG pipeline on the AI01 leaderboard.
leanlab is an open-source CLI tool designed for evolving and evaluating AI agent experiments and coding tasks. It supports experiment evolution against fixed metrics and automates the spec-to-merge workflow with locked acceptance tests, helping AI researchers and developers improve agent reliability and performance.
agent-skill-eval is an open-source CLI framework for evaluating the skills of code-generating agents across models like OpenCode, Claude Code, and Codex. It enables researchers and developers to benchmark agent performance using standardized tests.
Agent evaluation toolkit
tool-eval is a command-line framework for evaluating the tool usage of AI agents. It provides researchers and developers with tools to analyze and benchmark agent interactions with external tools.
AgentEval is an open-source .NET toolkit for evaluating AI agents. It provides features like tool usage validation, RAG quality metrics, stochastic evaluation, and model comparison, helping .NET developers assess and improve their AI agent implementations.
Zero-dependency eval harness for LLM and agent regression testing. Scores outputs with exact, contains, regex, JSON, citation, and token-F1 checks. Compares two runs to flag regressions.
primer-eval is an open-source CLI tool that provides a measurement harness for evaluating the context files used by AI coding agents. It is designed for AI researchers and developers working on agent performance and benchmarking.
