tea-tasting

tea-tasting is a Python package designed for the statistical analysis of A/B tests, offering a range of methods and tools tailored to experimenters and analysts working with experimental data. It addresses the challenges of running and interpreting A/B tests by providing built-in support for statistical techniques such as Welch's t-test, Student's t-test, z-test, Bootstrap, variance reduction using CUPED, the delta method for ratio metrics, and power analysis. The package also includes procedures for controlling multiple hypothesis testing, including false discovery rate (Benjamini-Hochberg and Benjamini-Yekutieli) and family-wise error rate (Hochberg's step-up and Holm's step-down). Users can analyze averages, proportions, ranks, quantiles, and ratios, as well as perform simulations like A/A tests and treatment simulations for power analysis.

tea-tasting is designed to work efficiently with a variety of data backends, reducing the need to manually aggregate or transfer large datasets. It supports input from platforms such as BigQuery, ClickHouse, PostgreSQL, Snowflake, and Trino through Ibis Tables, enabling users to perform calculations directly within their database engines. The package also accepts dataframes from Narwhals-supported libraries, including cuDF, Daft, Dask, DuckDB, Modin, pandas, Polars, PyArrow, and PySpark. For analyses that require granular data, such as Bootstrap, tea-tasting can fetch detailed records as needed.

The tool provides a convenient and extensible API that allows users to define custom metrics and apply statistical tests of their choice, aiming to minimize manual work and reduce the risk of errors. Its output is formatted for clarity, with features like rounding to significant digits and rendering results in various environments, including terminals, Jupyter/IPython, and marimo notebooks. Results can also be serialized to Markdown or converted to pandas and Polars DataFrames for further analysis or reporting.

Comprehensive documentation is available to guide users through its features and usage scenarios. tea-tasting is positioned as a specialized solution for those conducting and analyzing A/B tests, streamlining the statistical workflow from data access to result interpretation.