mcp-llm-eval

Purpose: Automating and standardizing LLM evaluation in CI/CD pipelines for developers and ML engineers.

Inferred · not functionally tested

Audience: machine learning engineers

Inferred · not functionally tested

Functions: monitoring, workflow_automation

Inferred · not functionally tested

Interfaces: API: indicated (inferred, not tested) · MCP: indicated (inferred, not tested) · CLI: indicated (inferred, not tested) · Self-hosting: indicated (inferred, not tested)

Recorded constraints: pricing: open_source · license: MIT · platforms: CLI · deployment: cli, api_only, self_hosted

Constraint provenance is unknown; confirm requirements with the publisher.

Record sources: pypi.org · github.com. These links do not verify the individual claims.

mcp-llm-eval sits in PulseGate's LLM evaluation & benchmarks category. Inferred · not functionally tested: It focuses on automating and standardizing LLM evaluation in CI/CD pipelines for developers and ML engineers. Inferred · not functionally tested: It is built as an open-source project for machine learning engineers. Basis unknown · not verified: The project is open source (MIT). Basis unknown · not verified: mcp-llm-eval is available on the command line and API, and it can be self-hosted.

It is developed by berkayildi, and it first shipped in 2026. The project is developed in the open on GitHub with 57 commits in the last 90 days. Inferred · not functionally tested: Key capabilities include LLM evaluation, CI/CD integration, and reusable gates. Inferred · not functionally tested: Catalogued interfaces include an MCP server and a public API.

Summary written by a language model from the project’s public pages.