judge-bench
judge-bench is an open-source package that provides synthetic bias and calibration probes for evaluating the reliability of LLMs acting as judges. It supports researchers in testing and analyzing model bias and calibration.
Overview
4 featuresIn the Model leaderboards space, judge-bench takes a focused approach. It enables systematic testing of LLM-as-judge reliability by generating synthetic bias and calibration probes. judge-bench is an open-source project aimed at AI evaluation researchers and developers. The project is open source (MIT). It ships for the command line, and it can be self-hosted.
It is developed by auraoneai, and it first shipped in 2026. The project is developed in the open on GitHub with 13 commits in the last 90 days. Key capabilities include bias probes, calibration probes, and LLM-as-judge testing.
Summary written by a language model from the project’s public pages.
- Bias probes
- Calibration probes
- LLM-as-judge testing
- Synthetic data generation
Built with & integrations
Trust & compliance
Indexing history
1What PulseGate has recorded for this listing
Frequently asked questions about judge-bench
- What does judge-bench do?
- Judge-bench enables systematic testing of LLM-as-judge reliability by generating synthetic bias and calibration probes. It is catalogued under Model leaderboards on PulseGate.
- Who is judge-bench for?
- judge-bench is an open-source project built for AI evaluation researchers and developers.
- Is judge-bench free?
- Yes — judge-bench is open source under the MIT license and free to use.
- What platforms does judge-bench run on?
- judge-bench runs on the command line. It can also be self-hosted.
- Is judge-bench still active?
- The GitHub repository shows 13 commits in the last 90 days.
- What are alternatives to judge-bench?
- Similar projects tracked by PulseGate include judgegate, journeyman-bench, and porchbench.judgegatejourneyman-benchporchbench
- Who makes judge-bench?
- judge-bench is developed by auraoneai.
- When did judge-bench launch?
- judge-bench first shipped in 2026.
Similar projects
Closest matches by what these projects do