Who is mcp-llm-eval for?

mcp-llm-eval is an open-source project built for machine learning engineers.

Is mcp-llm-eval free?

Yes — mcp-llm-eval is open source under the MIT license and free to use.

What platforms does mcp-llm-eval run on?

mcp-llm-eval runs on the command line and API. It can also be self-hosted.

Is mcp-llm-eval still active?

The GitHub repository shows 57 commits in the last 90 days.

What projects are similar to mcp-llm-eval?

Similar projects tracked by PulseGate include lm-mcp, llmesh-mcp, and llm-wiki-mcp.

Who develops mcp-llm-eval?

mcp-llm-eval is developed by berkayildi.

When did mcp-llm-eval launch?

mcp-llm-eval first shipped in 2026.

Back to the index

mcp-llm-eval

Name: mcp-llm-eval
Author: berkayildi

PyPIInfrastructure

mcp-llm-eval is an open-source CLI tool and MCP server that packages LLM evaluation gates as reusable primitives for CI/CD workflows. It enables developers and ML engineers to automate and standardize the evaluation of large language models within their development pipelines. The tool supports benchmarking and integration with MCP for scalable model assessment.

Open SourceMITCLIAPISelf-hosted

Visit PyPI

Overview

5 features

mcp-llm-eval sits in PulseGate's LLM eval & observability category. It focuses on automating and standardizing LLM evaluation in CI/CD pipelines for developers and ML engineers. It is built as an open-source project for machine learning engineers. The project is open source (MIT). mcp-llm-eval is available on the command line and API, and it can be self-hosted.

It is developed by berkayildi, and it first shipped in 2026. The project is developed in the open on GitHub with 57 commits in the last 90 days. Key capabilities include LLM evaluation, CI/CD integration, and reusable gates. It exposes integrations via an MCP server and a public API.

Summary written by a language model from the project’s public pages.

LLM evaluation
CI/CD integration
Reusable gates
MCP server
Benchmarking

Built with & integrations

AI providers

openai

Connectors

MCPAPI

Runs on

CLIAPI-onlySelf-hosted

Trust & compliance

License

MIT

Verified signals

HTTPSOpen SourceFree tierGitHubActive maintenance

Indexing history

What PulseGate has recorded for this listing

Indexed29 Jun · 11:45 UTC
Listing verified against its public source
Source: PulseGate · Open

Frequently asked questions about mcp-llm-eval

What is mcp-llm-eval?: Mcp-llm-eval focuses on automating and standardizing LLM evaluation in CI/CD pipelines for developers and ML engineers. It is catalogued under LLM eval & observability on PulseGate.
Who is mcp-llm-eval for?: mcp-llm-eval is an open-source project built for machine learning engineers.
Is mcp-llm-eval free?: Yes — mcp-llm-eval is open source under the MIT license and free to use.
What platforms does mcp-llm-eval run on?: mcp-llm-eval runs on the command line and API. It can also be self-hosted.
Is mcp-llm-eval still active?: The GitHub repository shows 57 commits in the last 90 days.
What projects are similar to mcp-llm-eval?: Similar projects tracked by PulseGate include lm-mcp, llmesh-mcp, and llm-wiki-mcp.lm-mcp llmesh-mcp llm-wiki-mcp
Who develops mcp-llm-eval?: mcp-llm-eval is developed by berkayildi.
When did mcp-llm-eval launch?: mcp-llm-eval first shipped in 2026.

At a glance

Platforms: Cli
Languages: English
Open source: Yes (GitHub)
License: MIT
Built for: machine learning engineers
Model: Open source
Solves: Automating and standardizing LLM evaluation in CI/CD pipelines for developers and ML engineers.

Registered as

GitHub: berkayildi/mcp-llm-eval
PyPI: mcp-llm-eval

Developer

berkayildi

Small team

GitHub

Open source

View on GitHub

Stars: 0
Forks: 0
Open issues: 0
Last commit: 3 May 2026
Commits 90d: 57
Contributors: 2
Authorship: Small team
Default branch: main
Latest release: v0.9.2 · 3 May 2026

Index record

Identity confidence: Low · 64
Indexed: 29 Jun 2026
Lifecycle: Alive
Last seen: 29 Jun 2026

Identity audit (12)

Slug: mcp-llm-eval-pypi-org
Lifecycle last checked: 8 Aug 2026
Verification state: Indexed for public listing
Listing state: Listed: yes
Index status: Included in index
Latest evidence snapshot: 29 Jun 2026
Timeline basis: Indexed-at chronology. This listing's first-seen date was written by the catalog backfill, not observed here, so it is not treated as a sighting.
Name from: Written by a language model from the project's public pages.
Category from: Assigned by a language model.
Summary from: Written by a language model from public pages.
Languages from: Detected by a language model from page content.
Canonical URL: https://pypi.org/project/mcp-llm-eval

Ship this? Send a correction — no account, and you get a link to follow it.

Similar projects

Closest matches by what these projects do