judgegate
judgegate is an open-source CLI tool for evaluating LLM judges in continuous integration workflows. It provides agreement statistics against human labels, bias probes, and a label budget calculator, supporting robust AI evaluation and monitoring for developers and researchers.
Overview
5 featuresjudgegate is a LLM evaluation & benchmarks project. It helps developers evaluate and monitor the reliability and bias of LLM judges in CI pipelines. judgegate is an open-source project aimed at AI evaluation engineers. The project is open source (MIT). It runs on the command line, and it can be self-hosted.
It is developed by yashchimata, and it first shipped in 2026. The project is developed in the open on GitHub with 1 commit in the last 90 days. Among its 5 catalogued features are agreement statistics, bias probes, and label budget calculator.
Summary written by a language model from the project’s public pages.
- Agreement statistics
- Bias probes
- Label budget calculator
- CI integration
- LLM judge evaluation
Built with & integrations
Trust & compliance
Indexing history
1What PulseGate has recorded for this listing
- Indexed15 Jul · 20:43 UTCjudgegate verified via PyPI Bulk EnumeratorSource: PyPI Bulk Enumerator · Open
Frequently asked questions about judgegate
- What is judgegate?
- Judgegate helps developers evaluate and monitor the reliability and bias of LLM judges in CI pipelines. It is catalogued under LLM evaluation & benchmarks on PulseGate.
- Who should use judgegate?
- judgegate is an open-source project built for AI evaluation engineers.
- Does judgegate have a free plan?
- Yes — judgegate is open source under the MIT license and free to use.
- What platforms does judgegate run on?
- judgegate runs on the command line. It can also be self-hosted.
- Is judgegate still active?
- The GitHub repository shows 1 commit in the last 90 days.
- Who develops judgegate?
- judgegate is developed by yashchimata.
- When did judgegate launch?
- judgegate first shipped in 2026.
- Is judgegate open source?
- Yes — judgegate is open source under the MIT license, developed on GitHub.
Similar projects
Closest matches by what these projects do