Skip to content
Back to the index

promptbench-cli

github.comInfrastructure

PulseGate's liveness check found it on 14 Sep 2026; it is registered on GitHub and PyPI and has been in the index since 17 Jun 2026. How this is checked

promptbench-cli is an open-source command-line tool that enables users to A/B test prompts across various large language model providers directly from their terminal. It supports multiple LLM APIs and is designed for developers and prompt engineers who need to benchmark and compare prompt outputs efficiently. The tool is MIT licensed and easy to install via PyPI.

Inferred · not functionally tested

Open SourceMITWebCLI
promptbench-cli preview
Visit github.com

Overview

5 features

Purpose: Comparing and benchmarking prompt performance across multiple LLM providers from the command line.

Inferred · not functionally tested

Audience: AI developers and prompt engineers

Inferred · not functionally tested

Functions: analytics

Inferred · not functionally tested

Interfaces: API: indicated (inferred, not tested) · MCP: unknown · CLI: indicated (inferred, not tested) · Self-hosting: unknown

Recorded constraints: pricing: open_source · license: MIT · platforms: CLI · deployment: browser, cli

Constraint provenance is unknown; confirm requirements with the publisher.

Record sources: github.com. These links do not verify the individual claims.

In the LLM evaluation & benchmarks space, promptbench-cli takes a focused approach. Inferred · not functionally tested: It focuses on comparing and benchmarking prompt performance across multiple LLM providers from the command line. Inferred · not functionally tested: It is built as an open-source project for AI developers and prompt engineers. Basis unknown · not verified: promptbench-cli is open source under the MIT license. Basis unknown · not verified: It ships for the web and the command line.

Shehriyar Ali Rustam builds and maintains promptbench-cli, and it first shipped in 2026. Inferred · not functionally tested: Key capabilities include prompt benchmarking, A/B testing, and LLM provider support. Inferred · not functionally tested: Catalogued interfaces include a public API.

Summary written by a language model from the project’s public pages.

Tasks: Inferred · not functionally tested

  • Prompt benchmarking
  • A/B testing
  • LLM provider support
  • Terminal interface
  • MIT license

Topics: Inferred · not functionally tested

Tags
prompt-ab-testingllm-benchmarkingcli-tool
AI capabilities
Text

JSON profile · Text profile · Access guide

Built with & integrations

AI providers
openaianthropicgoogle_gemini
Connectors
API
Runs on
BrowserCLI

Trust & compliance

License
MIT
Public signals
HTTPSOpen SourceFree tierGitHub

Indexing history

What PulseGate has recorded for this listing

Nothing recorded for this listing in this window.

Frequently asked questions about promptbench-cli

What is promptbench-cli?
Inferred · not functionally tested: Promptbench-cli focuses on comparing and benchmarking prompt performance across multiple LLM providers from the command line. It is catalogued under LLM evaluation & benchmarks on PulseGate.
Who should use promptbench-cli?
Inferred · not functionally tested: promptbench-cli is an open-source project built for AI developers and prompt engineers.
Is promptbench-cli free?
Basis unknown · not verified: Yes — promptbench-cli is open source under the MIT license and free to use.
What platforms does promptbench-cli run on?
Basis unknown · not verified: promptbench-cli runs on the web and the command line.
Is promptbench-cli still active?
PulseGate's liveness check found it on 14 Sep 2026.
What are alternatives to promptbench-cli?
Similar projects tracked by PulseGate include promptlint-cli, ai-prompt-forge, and litebench.promptlint-cliai-prompt-forgelitebench
Who makes promptbench-cli?
promptbench-cli is developed by Shehriyar Ali Rustam.
How long has promptbench-cli been around?
promptbench-cli first shipped in 2026.

Similar projects

Closest matches by what these projects do