bench-my-llm
bench-my-llm is an open-source command-line tool for benchmarking OpenAI-compatible language model APIs. It measures time to first token, tokens per second, latency, cost, and output quality for developers evaluating LLM services.
Overview
8 featuresbench-my-llm sits in PulseGate's LLM evaluation & benchmarks category. It focuses on comparing LLM API performance, cost, latency, and output quality without building custom benchmarking scripts. bench-my-llm is an open-source project aimed at developers and ML engineers. bench-my-llm is open source under the MIT license. bench-my-llm is available on the command line and API.
It is developed by Manas Vardhan, and it first shipped in 2026. The project is developed in the open on GitHub with 58 stars and 11 commits in the last 90 days. Among its 8 catalogued features are TTFT measurement, TPS measurement, and latency tracking. It exposes integrations via a public API.
Summary written by a language model from the project’s public pages.
- TTFT measurement
- TPS measurement
- Latency tracking
- Cost analysis
- Quality evaluation
- API benchmarking
- CLI interface
- OpenAI compatibility
Built with & integrations
- Unspecified agent
- AGENTS.md
Trust & compliance
Indexing history
1What PulseGate has recorded for this listing
Frequently asked questions about bench-my-llm
- What does bench-my-llm do?
- Bench-my-llm focuses on comparing LLM API performance, cost, latency, and output quality without building custom benchmarking scripts. It is catalogued under LLM evaluation & benchmarks on PulseGate.
- Who is bench-my-llm for?
- bench-my-llm is an open-source project built for developers and ML engineers.
- Does bench-my-llm have a free plan?
- Yes — bench-my-llm is open source under the MIT license and free to use.
- What platforms does bench-my-llm run on?
- bench-my-llm runs on the command line and API.
- Is bench-my-llm still maintained?
- The GitHub repository shows 11 commits in the last 90 days.
- What are alternatives to bench-my-llm?
- Similar projects tracked by PulseGate include llm-benchmark-runner, BenchLoop, and porchbench.llm-benchmark-runnerBenchLoopporchbench
- Who develops bench-my-llm?
- bench-my-llm is developed by Manas Vardhan.
- How long has bench-my-llm been around?
- bench-my-llm first shipped in 2026.
Similar projects
Closest matches by what these projects do