falsify

Falsify is a command-line tool designed to support pre-registration and continuous integration workflows for AI and machine learning claims, with a focus on compliance with the EU AI Act high-risk obligations, including Article 12 logging requirements. The tool enables users to declare and lock evaluation claims before running experiments, ensuring that thresholds and criteria are registered and cannot be altered after results are observed. This approach is intended to make claims auditable and prevent silent rewriting of evaluation criteria, thereby supporting evidence-grade compliance and reproducibility.

The workflow in Falsify consists of several stages: users declare a claim, lock it by generating a SHA-256 hash over a canonical YAML specification, and then execute the experiment according to the pre-registered test plan, metric, and threshold. The tool enforces that vague or incomplete specifications are rejected at lock time. During execution, Falsify verifies that the spec hash remains unchanged, and any modification after locking is detected as tampering. The outcome is a numeric verdict: exit code 0 indicates a pass (the result meets the locked threshold), exit code 10 indicates failure, and exit code 3 signals tampering, which causes CI systems to block further actions. Additionally, a git commit-msg hook is provided to scan verdict logs and prevent commits or documentation updates that contradict recorded results.

Falsify is implemented with reference code in Python, JavaScript, Go, and Rust, and is delivered as a package installable via pip. The tool uses a canonical YAML format for specifications, which can be validated in IDEs using SchemaStore. 0 license. The tool is suitable for compliance leads, researchers, and practitioners who need to ensure that AI evaluation claims are transparent, reproducible, and compliant with regulatory standards. Falsify's design emphasizes tamper detection, auditability, and rigorous enforcement of pre-registered experimental criteria.