Who should use AI Evaluator?

AI Evaluator is an open-source project built for AI developers building LLM agents.

Is AI Evaluator free?

Yes — AI Evaluator is open source under the MIT license and free to use.

What platforms does AI Evaluator run on?

AI Evaluator runs on the web and the command line. It can also be self-hosted.

Is AI Evaluator still maintained?

The GitHub repository shows 111 commits in the last 90 days.

What are alternatives to AI Evaluator?

Similar projects tracked by PulseGate include AI-Evals, ai-eval-forge, and evalite.

Who makes AI Evaluator?

AI Evaluator is developed by aievaluator-dev.

When did AI Evaluator launch?

AI Evaluator first shipped in 2026.

Back to the index

AI Evaluator

Name: AI Evaluator
Author: aievaluator-dev

aievaluator.devInfrastructure

AI Evaluator is an MIT-licensed command-line tool for evaluating LLM agents. It supports agent testing and evaluation workflows that can run locally or in CI/CD environments, targeting developers building AI applications.

Open SourceMITWebCLISelf-hosted

Visit aievaluator.dev

Overview

5 features

AI Evaluator is a LLM eval & observability project. It focuses on testing and measuring the quality of LLM agents from the command line and in CI/CD pipelines. AI Evaluator is an open-source project aimed at AI developers building LLM agents. The project is open source (MIT). It ships for the web and the command line, and it can be self-hosted.

aievaluator-dev builds and maintains AI Evaluator, and it first shipped in 2026. The project is developed in the open on GitHub with 111 commits in the last 90 days. Among its 5 catalogued features are LLM evaluation, agent testing, and CLI workflows.

Summary written by a language model from the project’s public pages.

LLM evaluation
Agent testing
CLI workflows
CI/CD integration
Evaluation reports

Built with & integrations

Connectors

github

Runs on

BrowserCLISelf-hosted

Trust & compliance

License

MIT

Verified signals

HTTPSOpen SourceFree tierGitHubActive maintenance

Indexing history

What PulseGate has recorded for this listing

Indexed7 Aug · 15:52 UTC
aievaluator verified against its public source
Source: PulseGate · Open

Frequently asked questions about AI Evaluator

What does AI Evaluator do?: AI Evaluator focuses on testing and measuring the quality of LLM agents from the command line and in CI/CD pipelines. It is catalogued under LLM eval & observability on PulseGate.
Who should use AI Evaluator?: AI Evaluator is an open-source project built for AI developers building LLM agents.
Is AI Evaluator free?: Yes — AI Evaluator is open source under the MIT license and free to use.
What platforms does AI Evaluator run on?: AI Evaluator runs on the web and the command line. It can also be self-hosted.
Is AI Evaluator still maintained?: The GitHub repository shows 111 commits in the last 90 days.
What are alternatives to AI Evaluator?: Similar projects tracked by PulseGate include AI-Evals, ai-eval-forge, and evalite.AI-Evals ai-eval-forge evalite
Who makes AI Evaluator?: AI Evaluator is developed by aievaluator-dev.
When did AI Evaluator launch?: AI Evaluator first shipped in 2026.

At a glance

Platforms: Cli · Web
Languages: English
Open source: Yes (GitHub)
License: MIT
First seen: 7 Aug 2026
Built for: AI developers building LLM agents
Model: Open source
Solves: Testing and measuring the quality of LLM agents from the command line and in CI/CD pipelines.

Registered as

GitHub: aievaluator-dev/aievaluator-cli
PyPI: aievaluator

Developer

aievaluator-dev

Solo developer

GitHub

Open source

View on GitHub

Stars: 0
Forks: 0
Open issues: 0
Last commit: 7 Aug 2026
Commits 90d: 111
Contributors: 1
Authorship: Solo
Default branch: master

Index record

Identity confidence: Low · 64
Indexed: 7 Aug 2026
Lifecycle: Alive
First seen: Aug 2026
Last seen: 7 Aug 2026

Identity audit (13)

Slug: aievaluator-pypi-org
Lifecycle last checked: 9 Aug 2026
Verification state: Indexed for public listing
Listing state: Listed: yes
Index status: Included in index
Latest evidence snapshot: 7 Aug 2026
Timeline basis: Indexed-at chronology (no inferred launch/funding milestones).
Name from: Written by a language model from the project's public pages.
Category from: Assigned by a language model.
Summary from: Written by a language model from public pages.
Languages from: Detected by a language model from page content.
Last updated: 7 Aug 2026
Canonical URL: https://aievaluator.dev

Ship this? Send a correction — no account, and you get a link to follow it.

Similar projects

Closest matches by what these projects do