synthetic-disagreement
synthetic-disagreement is an open-source Python package for generating deterministic synthetic reviewer disagreement and inter-annotator agreement stress curves. It aids researchers in evaluating annotation quality and simulating disagreement scenarios for robust evaluation.
Open SourceMITCLISelf-hosted
Overview
4 featuresIn the LLM eval & observability space, synthetic-disagreement takes a focused approach. It helps researchers simulate and analyze reviewer disagreement and annotation quality in evaluation workflows. It is built as an open-source project for AI evaluation researchers and data scientists. The project is open source (MIT). synthetic-disagreement is available on the command line, and it can be self-hosted.
It is developed by auraoneai, and it first shipped in 2026. Development happens publicly on GitHub with 12 commits in the last 90 days. Key capabilities include reviewer disagreement simulation, inter-annotator agreement, and stress curve generation.
Summary written by a language model from the project’s public pages.
- Reviewer disagreement simulation
- Inter-annotator agreement
- Stress curve generation
- Synthetic data
Tags
reviewer-disagreementannotation-qualitysynthetic-datainter-annotator-agreement
AI capabilities
Weights: Open
Built with & integrations
Runs on
CLISelf-hosted
Trust & compliance
License
MIT
Verified signals
Indexing history
1What PulseGate has recorded for this listing
- Indexed13 Jul · 23:00 UTCsynthetic-disagreement verified against its public sourceSource: PulseGate · Open
Frequently asked questions about synthetic-disagreement
- What is synthetic-disagreement?
- Synthetic-disagreement helps researchers simulate and analyze reviewer disagreement and annotation quality in evaluation workflows. It is catalogued under LLM eval & observability on PulseGate.
- Who should use synthetic-disagreement?
- synthetic-disagreement is an open-source project built for AI evaluation researchers and data scientists.
- Does synthetic-disagreement have a free plan?
- Yes — synthetic-disagreement is open source under the MIT license and free to use.
- What platforms does synthetic-disagreement run on?
- synthetic-disagreement runs on the command line. It can also be self-hosted.
- Is synthetic-disagreement still maintained?
- The GitHub repository shows 12 commits in the last 90 days.
- What are alternatives to synthetic-disagreement?
- Similar projects tracked by PulseGate include constraintloop, trajectory-judge, and DeFake.constraintlooptrajectory-judgeDeFake
- Who makes synthetic-disagreement?
- synthetic-disagreement is developed by auraoneai.
- How long has synthetic-disagreement been around?
- synthetic-disagreement first shipped in 2026.
Also in LLM eval & observability
Same category — not a similarity match