porchbench
PulseGate's liveness check found it on 14 Sep 2026; it is registered on GitHub and PyPI and has been in the index since 1 Jul 2026. How this is checked
porchbench is an open-source CLI tool designed for rigorous benchmarking and evaluation of local large language models (LLMs). It provides paired statistics, LLM-as-judge scoring, and ensures reproducible runs, making it ideal for AI researchers and developers who need to assess the quality and performance of local AI models. The tool supports integration with local inference engines such as Ollama and quantized models.
Inferred · not functionally tested
Overview
6 featuresPurpose: Enables developers to rigorously benchmark and evaluate the quality of local LLMs with reproducible and automated scoring.
Inferred · not functionally tested
Audience: AI researchers and developers working with local language models
Inferred · not functionally tested
Functions: analytics
Inferred · not functionally tested
Interfaces: API: unknown · MCP: unknown · CLI: indicated (inferred, not tested) · Self-hosting: indicated (inferred, not tested)
Recorded constraints: pricing: open_source · license: Apache-2.0 · platforms: CLI · deployment: cli, self_hosted
Constraint provenance is unknown; confirm requirements with the publisher.
Record sources: pypi.org · github.com. These links do not verify the individual claims.
porchbench is a LLM evaluation & benchmarks project. Inferred · not functionally tested: It enables developers to rigorously benchmark and evaluate the quality of local LLMs with reproducible and automated scoring. Inferred · not functionally tested: It is built as an open-source project for AI researchers and developers working with local language models. Basis unknown · not verified: porchbench is open source under the Apache-2.0 license. Basis unknown · not verified: It runs on the command line, and it can be self-hosted.
Behind porchbench is mmdevelops, and it first shipped in 2026. Development happens publicly on GitHub with 138 commits in the last 90 days. Inferred · not functionally tested: Among its 6 catalogued features are paired statistics, LLM-as-judge scoring, and reproducible runs.
Summary written by a language model from the project’s public pages.
Tasks: Inferred · not functionally tested
- Paired statistics
- LLM-as-judge scoring
- Reproducible runs
- Local LLM support
- Quantization support
- Routing evaluation
Topics: Inferred · not functionally tested
Built with & integrations
Trust & compliance
Indexing history
What PulseGate has recorded for this listing
Frequently asked questions about porchbench
- What is porchbench?
- Inferred · not functionally tested: Porchbench enables developers to rigorously benchmark and evaluate the quality of local LLMs with reproducible and automated scoring. It is catalogued under LLM evaluation & benchmarks on PulseGate.
- Who should use porchbench?
- Inferred · not functionally tested: porchbench is an open-source project built for AI researchers and developers working with local language models.
- Is porchbench free?
- Basis unknown · not verified: Yes — porchbench is open source under the Apache-2.0 license and free to use.
- What platforms does porchbench run on?
- Basis unknown · not verified: porchbench runs on the command line. It can also be self-hosted.
- Is porchbench still active?
- PulseGate's liveness check found it on 14 Sep 2026. Its GitHub repository shows 138 commits in the last 90 days.
- What are alternatives to porchbench?
- Similar projects tracked by PulseGate include bench-my-llm, BenchLoop, and benchrig.bench-my-llmBenchLoopbenchrig
- Who makes porchbench?
- porchbench is developed by mmdevelops.
- When did porchbench launch?
- porchbench first shipped in 2026.
Similar projects
Closest matches by what these projects do