local-bench-ai is a benchmarking and evaluation tool designed for comparing the performance and quality of open-weight AI models running on local hardware. It addresses the need for standardized, reproducible evaluation of local inference setups, focusing on how different models and quantization variants perform across a range of VRAM capacities and hardware configurations. The tool features a public leaderboard that ranks models and their variants using a Local Intelligence Index, which aggregates scores across axes such as agentic ability, knowledge, instruction following, tool use, coding, and math. Each leaderboard entry includes detailed metrics like tokens per second, VRAM usage, latency, accuracy, and total benchmarking time, enabling users to compare models based on both quality and hardware efficiency.
Users can select a model, choose VRAM and runtime settings, and run benchmarks using precise commands provided by local-bench-ai. The tool supports benchmarking on various VRAM tiers, from 2 GB up to 512 GB, and provides guidance for running both public static-path tests (covering five axes) and full six-axis evaluations, the latter requiring a managed AppWorld harness and specific configuration. 11+ and llama-server, and it can be installed via pip using the package name local-bench-ai. The tool verifies model downloads against pinned hashes and checks publishability before benchmarking or submitting results.
Advanced users have the option to bring their own OpenAI-compatible server or use custom hardware setups, with support for vLLM and explicit publishable metadata. The tool also integrates with Hugging Face repositories for model downloads, handling gated repos and authentication where necessary. While public benchmarks are run with the --static-only flag, full six-axis runs require additional setup and are eligible for the comprehensive Local Intelligence Index ranking. The platform emphasizes reproducibility by generating run hashes and recording detailed provenance for each benchmark.
local-bench-ai is positioned as a specialized AI evaluation and observability tool for practitioners seeking transparent, hardware-aware benchmarking of local AI models.
In the LLM eval & observability space, local-bench-ai takes a focused approach. It focuses on comparing and benchmarking the performance of local AI models and setups in a standardized way. It is built as an open-source project for ai researchers. local-bench-ai is open source under the Apache-2.0 license. local-bench-ai is available on the web and the command line.
It is developed by local-bench, and the product first shipped in 2026. Development happens publicly on GitHub with 9 commits in the last 90 days. PulseGate's similarity index finds few close equivalents — local-bench-ai occupies a relatively distinct niche. Key capabilities include benchmark runner, leaderboard integration, and Local AI evaluation.
Latest indexed changes and source events
local-bench-ai discovered by the PulseGate indexer
Other apps tracked under the same category.
