assertllm

assertllm is a pytest-style tool for testing LLM outputs like ordinary code. It is aimed at developers who want to check AI responses with assertions rather than manual review, and its site describes it as a way to test AI outputs like code.

Its core features include more than 22 built-in assertions covering text, JSON, structured output, performance, and agent behavior. The examples and feature list show assertions such as contains, matches_regex, valid_json, structured_output, latency_under, cost_under, tool_called, no_loop, and support for call ordering. It also supports composable logic with AND, OR, and custom combinations using & and | operators. The tool states that most assertions are deterministic and instant, and that no LLM judge is needed for most checks. Built-in retry support is included at both the decorator and fixture level for handling non-deterministic outputs.

assertllm is built on Pydantic, with models using BaseModel for auto-validation, JSON serialization, and schema generation. It supports OpenAI, Anthropic, and Ollama out of the box, and says that only the needed integrations can be installed. The page also shows an example of use with @llm_test and a model argument, alongside terminal output from a test run.

It is distributed as open source under the MIT license, and the page shows a pip installation command. The product is presented as version 0.1.0.