Coder Eval
PulseGate's liveness check found it on 5 Oct 2026; it has been in the index since 4 Sep 2026. How this is checked
Coder Eval is an open-source CLI framework for evaluating Claude Code skills, MCP servers, and command-line tools. It supports sandboxed YAML test suites, activation checks, A/B experiments, and CI quality gates for developers building agent workflows.
Inferred · not functionally tested
Overview
6 featuresPurpose: Testing whether agent skills, MCP servers, and CLIs work reliably when used by AI agents.
Inferred · not functionally tested
Audience: AI developers and evaluation engineers
Inferred · not functionally tested
Functions: agents, analytics, monitoring
Inferred · not functionally tested
Interfaces: API: unknown · MCP: indicated (inferred, not tested) · CLI: indicated (inferred, not tested) · Self-hosting: indicated (inferred, not tested)
Recorded constraints: pricing: open_source · license: Open Source · platforms: CLI · deployment: cli, self_hosted
Constraint provenance is unknown; confirm requirements with the publisher.
Record sources: github.com. These links do not verify the individual claims.
Coder Eval sits in PulseGate's Agent evaluation & testing category. Inferred · not functionally tested: It focuses on testing whether agent skills, MCP servers, and CLIs work reliably when used by AI agents. Inferred · not functionally tested: Coder Eval is an open-source project aimed at AI developers and evaluation engineers. Basis unknown · not verified: Coder Eval is open source under the Open Source license. Basis unknown · not verified: It ships for the command line, and it can be self-hosted.
Behind Coder Eval is UiPath. Inferred · not functionally tested: Among its 6 catalogued features are sandboxed test suites, YAML configuration, and activation checks. Basis unknown · not verified: Catalogued interfaces include an MCP server.
Summary written by a language model from the project’s public pages.
Tasks: Inferred · not functionally tested
- Sandboxed test suites
- YAML configuration
- Activation checks
- A/B experiments
- CI gates
- MCP testing
Topics: Inferred · not functionally tested
Built with & integrations
- anthropic
- bclaude in the HTML · bclaude- in the HTML
Trust & compliance
Indexing history
1What PulseGate has recorded for this listing
- Indexed4 Sep · 23:17 UTCShow HN: Coder Eval – A Framework for Evals seen via Hacker News firehose (Algolia)Source: Hacker News firehose (Algolia) · Open
Frequently asked questions about Coder Eval
- What is Coder Eval?
- Inferred · not functionally tested: Coder Eval focuses on testing whether agent skills, MCP servers, and CLIs work reliably when used by AI agents. It is catalogued under Agent evaluation & testing on PulseGate.
- Who should use Coder Eval?
- Inferred · not functionally tested: Coder Eval is an open-source project built for AI developers and evaluation engineers.
- Does Coder Eval have a free plan?
- Basis unknown · not verified: Yes — Coder Eval is open source under the Open Source license and free to use.
- What platforms does Coder Eval run on?
- Basis unknown · not verified: Coder Eval runs on the command line. It can also be self-hosted.
- Is Coder Eval still active?
- PulseGate's liveness check found it on 5 Oct 2026.
- What are alternatives to Coder Eval?
- Similar projects tracked by PulseGate include coder-eval, caliper-eval, and agent-skill-eval.coder-evalcaliper-evalagent-skill-eval
- Who develops Coder Eval?
- Coder Eval is developed by UiPath.
- Is Coder Eval open source?
- Basis unknown · not verified: Yes — Coder Eval is open source under the Open Source license.
Similar projects
Closest matches by what these projects do