porchbench
porchbench is an open-source CLI tool designed for rigorous benchmarking and evaluation of local large language models (LLMs). It provides paired statistics, LLM-as-judge scoring, and ensures reproducible runs, making it ideal for AI researchers and developers who need to assess the quality and performance of local AI models. The tool supports integration with local inference engines such as Ollama and quantized models.
Open SourceApache-2.0CLISelf-hosted
Overview
6 featuresporchbench is a LLM eval & observability project. It enables developers to rigorously benchmark and evaluate the quality of local LLMs with reproducible and automated scoring. It is built as an open-source project for AI researchers and developers working with local language models. porchbench is open source under the Apache-2.0 license. It runs on the command line, and it can be self-hosted.
Behind porchbench is mmdevelops, and it first shipped in 2026. Development happens publicly on GitHub with 138 commits in the last 90 days. Among its 6 catalogued features are paired statistics, LLM-as-judge scoring, and reproducible runs.
Summary written by a language model from the project’s public pages.
- Paired statistics
- LLM-as-judge scoring
- Reproducible runs
- Local LLM support
- Quantization support
- Routing evaluation
Tags
llm-benchmarkinglocal-llmmodel-evaluationllm-as-judgereproducible-eval
AI capabilities
Text
Inference: Local
Built with & integrations
AI providers
anthropiclocal_oss
Runs on
CLISelf-hosted
Trust & compliance
License
Apache-2.0
Verified signals
Indexing history
1What PulseGate has recorded for this listing
Frequently asked questions about porchbench
- What is porchbench?
- Porchbench enables developers to rigorously benchmark and evaluate the quality of local LLMs with reproducible and automated scoring. It is catalogued under LLM eval & observability on PulseGate.
- Who should use porchbench?
- porchbench is an open-source project built for AI researchers and developers working with local language models.
- Is porchbench free?
- Yes — porchbench is open source under the Apache-2.0 license and free to use.
- What platforms does porchbench run on?
- porchbench runs on the command line. It can also be self-hosted.
- Is porchbench still active?
- The GitHub repository shows 138 commits in the last 90 days.
- What are alternatives to porchbench?
- Similar projects tracked by PulseGate include BenchLoop, homebench, and llm-benchmark-runner.BenchLoophomebenchllm-benchmark-runner
- Who makes porchbench?
- porchbench is developed by mmdevelops.
- When did porchbench launch?
- porchbench first shipped in 2026.
Similar projects
Closest matches by what these projects do