llm-canary
llm-canary is an open-source regression testing tool designed for developers building LLM-powered applications. It enables YAML-based test suites, offline-first assertions, baseline diffing, and agent-trace policy gates, making it easy to automate evaluation and catch regressions in CI pipelines. Ideal for teams seeking robust LLM app quality assurance.
Overview
7 featuresllm-canary is a LLM eval & observability project. It focuses on ensuring LLM app reliability by automating regression testing and baseline comparisons in CI workflows. llm-canary is an open-source project aimed at LLM app developers. The project is open source (MIT). It runs on the command line.
Behind llm-canary is okssusucha, and it first shipped in 2026. Development happens publicly on GitHub with 8 commits in the last 90 days. Key capabilities include YAML test suites, offline assertions, and baseline diffing.
Summary written by a language model from the project’s public pages.
- YAML test suites
- Offline assertions
- Baseline diffing
- Agent-trace policy gates
- CI integration
- Regression testing
- Prompt evaluation
Built with & integrations
Trust & compliance
Indexing history
1What PulseGate has recorded for this listing
Frequently asked questions about llm-canary
- What is llm-canary?
- Llm-canary focuses on ensuring LLM app reliability by automating regression testing and baseline comparisons in CI workflows. It is catalogued under LLM eval & observability on PulseGate.
- Who is llm-canary for?
- llm-canary is an open-source project built for LLM app developers.
- Does llm-canary have a free plan?
- Yes — llm-canary is open source under the MIT license and free to use.
- What platforms does llm-canary run on?
- llm-canary runs on the command line.
- Is llm-canary still active?
- The GitHub repository shows 8 commits in the last 90 days.
- What are alternatives to llm-canary?
- Similar projects tracked by PulseGate include Red Teamer Intuition Quotient, agentharness-verifier, and tokenmaxxing.Red Teamer Intuition Quotientagentharness-verifiertokenmaxxing
- Who makes llm-canary?
- llm-canary is developed by okssusucha.
- When did llm-canary launch?
- llm-canary first shipped in 2026.
Also in LLM eval & observability
Same category — not a similarity match