Skip to content
Alternatives
Software like llm-agent-bench
What else does this job. Matched on what each project does, not on who links to whom.
Closest first
- litebenchgithub.comlitebench is an open-source CLI tool that enables developers and researchers to benchmark large language models and AI agents. It supports quick setup and evaluation workflows, including popular benchmarks like GSM8K and HumanEval.
- agentbench-clipypi.orgagentbench-cli is an open-source command-line tool that allows AI developers and researchers to evaluate, test, and scan the behavior and safety of AI agents. It provides automated checks and reporting for agent performance and compliance.
- llm-parliamentpypi.orgllm-parliament is an open-source CLI tool that facilitates multi-agent debate among AI models to improve decision-making. It is research-backed and designed for local-first operation, targeting AI researchers and developers interested in agent-based reasoning.
- benchflowgithub.comMulti-turn agent benchmarking with ACP — run any agent, any model, any provider.
- local-bench-ailocal-bench.ailocal-bench-ai is a local AI benchmarking suite for open-weight models running on local hardware. It compares model quality against the amount of VRAM required to run them, and it presents results as a Local Intelligence Index with scored variants on a board and leaderboard. The suite measures tokens per second, VRAM, latency, accuracy, and the reproducible run hash attached to each artifact. It supports a local, open-weight, judge-free runtime with deterministic warmup prompts, answer-key evaluation, and separate public and full execution paths. The public command runs five non-agentic axes with a static-only mode, while full six-axis execution uses a managed AppWorld harness. The leaderboard shows ranked model variants, including per-axis scores for agentic ability, knowledge, instruction following, tool calling, coding, and math, along with bench time and fit by VRAM tier. Users can pick a model, select VRAM and runtime settings, and get exact commands for benchmarking. The page also shows a catalog browse view and a way to paste a Hugging Face repository. Runtime options named on the page include llama.cpp, LM Studio, and vLLM, and it notes support for bringing a custom OpenAI-compatible server or custom hardware setups. The commands verify downloads against pinned hashes, cache tokenizers, check publishability before starting, and can ask before submitting results. The page also records tokenizer and chat-template digests for some runs. Installation is shown with pip using the package name local-bench-ai, including an extra for Hugging Face support. The page warns that plain pip install localbench installs an unrelated third-party package. It also states that Python 3.11+ is required for the public static path, and that the full six-axis lane additionally needs Windows with WSL2. Hugging Face gated repositories are supported through hf auth login after license acceptance, and llama-server can read its token from the HF_TOKEN environment variable. The package is described as a catalog-pinned one-command flow for benchmark, score, and optional submit workflows.
- llm-bench-costgithub.comCLI tool for comparing LLM API pricing, ranked by cost-effectiveness against LMSYS Arena scores
- benchskillspypi.orgbenchskills is an open-source framework for benchmarking and evaluating the skills of AI agents. It provides tools for researchers and developers to assess agent performance across various tasks and metrics.
- agent-safety-bench-envspypi.orgagent-safety-bench-envs is an open-source package providing tool environments for the Agent Safety Bench (ASB) benchmark. It enables researchers and developers to evaluate and test the safety of AI agents using customizable scenarios and CLI tools.
- agentlintpypi.orgagentlint is an open-source CLI tool that provides real-time quality guardrails and linting for AI coding agents. It helps developers maintain code quality and safety by integrating with agentic coding workflows and enforcing best practices.
- BenchLLMbenchllm.comBenchLLM is a platform designed for evaluating large language model (LLM) applications. It enables developers to build test suites, generate quality reports, and choose between automated, interactive, or custom evaluation strategies. BenchLLM supports both API and CLI usage, making it suitable for AI developers and ML engineers seeking robust model evaluation tools.
- llm-benchmark-runnergithub.comllm-benchmark-runner is an open-source CLI tool that allows developers and researchers to benchmark the inference latency and throughput of large language model APIs. It supports OpenAI-compatible endpoints and provides detailed performance metrics for model evaluation.
- AI Evaluatoraievaluator.devAI Evaluator is an MIT-licensed command-line tool for evaluating LLM agents. It supports agent testing and evaluation workflows that can run locally or in CI/CD environments, targeting developers building AI applications.
- lambda-agentgithub.comlambda-agent is an open-source minimal AI coding agent that automates code generation tasks through a command-line interface. It is designed for developers seeking lightweight AI-powered coding assistance.
- Terminal-Benchtbench.aiTerminal-Bench provides a suite of benchmarks for evaluating the capabilities of AI agents in terminal environments. It offers standardized tasks, leaderboards, and performance metrics to help researchers and developers assess and compare agent performance. The platform is open source and designed for the AI research community.
- agentanvilgithub.comContract-based testing framework for LLM agents — hybrid metrics (objective + LLM-as-judge + human), multi-agent and A2A protocol support, deterministic record/replay envelope.
- agenthub-pythonpypi.orgagenthub-python is a Python SDK and API hub designed for the agent era, providing unified access to multiple large language models including Gemini, Claude, and GPT. It focuses on high-precision capabilities needed for building reliable autonomous agents. The open-source package (Apache-2.0) simplifies LLM orchestration, routing, and agent tooling for AI developers.
- pawbenchgithub.com4-dimensional LLM inference benchmark — multi-turn, multi-agent, parallel dispatch with tool calling
- CooperBenchcooperbench.comCooperBench is a benchmarking platform designed to evaluate the performance and cooperation of AI agent teams on collaborative coding tasks. It addresses the challenge of measuring how well AI agents can function as teammates, especially in scenarios where tasks are divided between agents and require coordination to avoid conflicts. The benchmark aims to inform research on both human-AI and human-agent collaboration by highlighting the coordination deficits that currently limit the effectiveness of AI agent teams. The platform includes a suite of 652 tasks sourced from 12 widely used open-source repositories across four programming languages: Python, TypeScript, Go, and Rust. Each task is constructed so that two agents are assigned different features to implement, which can be worked on independently but may conflict if not properly coordinated. The tasks, along with accompanying unit tests and ground-truth code, were created by a team of eight annotators with real-world software engineering experience. CooperBench provides a leaderboard that ranks models based on their cooperative success rate, allowing for comparison of agent team performance across models and task difficulties. Key findings from using CooperBench reveal that AI agents working together as a team perform significantly worse than when a single agent handles both parts of a task, with success rates dropping by approximately 30% or more in team settings. The benchmark also analyzes the nature of agent coordination failures, identifying expectation failures, communication failures, and commitment failures as primary causes. Despite the presence of real-time communication channels and the ability for agents to negotiate roles or divide resources, these issues persist, underscoring fundamental barriers to effective agent cooperation. The platform offers tools for viewing agent interaction logs and analyzing emergent coordination patterns among successful and failed runs. CooperBench can be installed via Python package managers such as pip, and its code and dataset are available on GitHub and HuggingFace. The project is a collaboration between Stanford University and SAP Labs US.
Ranked by how close each one sits to llm-agent-bench in the index, not by popularity. Back to llm-agent-bench →