Simple A/B Test Example¶

This notebook demonstrates a basic A/B test analysis using cluster-experiments.

Overview¶

We'll simulate an experiment where we test a new feature's impact on:

• Conversions (simple metric): Whether a user made a purchase
• Conversion Rate (ratio metric): Conversions per visit
• Revenue (simple metric): Total revenue generated

Setup¶

In [3]:

import pandas as pd
import numpy as np
import warnings
warnings.filterwarnings('ignore')

from cluster_experiments import AnalysisPlan

# random seed
np.random.seed(42)

import pandas as pd import numpy as np import warnings warnings.filterwarnings('ignore') from cluster_experiments import AnalysisPlan # random seed np.random.seed(42)

1. Generate Simulated Experiment Data¶

Let's create a dataset with control and treatment groups.

In [4]:

n_users = 2000

# Create base data
data = pd.DataFrame({
    'user_id': range(n_users),
    'variant': np.random.choice(['control', 'treatment'], n_users),
    'visits': np.random.poisson(10, n_users),  # Number of visits
})

# Simulate conversions (more likely for treatment)
data['converted'] = (
    np.random.binomial(1, 0.10, n_users) |  # Base conversion rate
    (data['variant'] == 'treatment') & np.random.binomial(1, 0.03, n_users)  # +3% for treatment
).astype(int)

# Simulate revenue (higher for converters and treatment)
data['revenue'] = 0.0
converters = data['converted'] == 1
data.loc[converters, 'revenue'] = np.random.gamma(shape=2, scale=25, size=converters.sum())

# Treatment group gets slightly higher revenue
treatment_converters = (data['variant'] == 'treatment') & converters
data.loc[treatment_converters, 'revenue'] *= 1.15

print(f"Dataset shape: {data.shape}")
print(f"\nFirst few rows:")
data.head(10)

n_users = 2000 # Create base data data = pd.DataFrame({ 'user_id': range(n_users), 'variant': np.random.choice(['control', 'treatment'], n_users), 'visits': np.random.poisson(10, n_users), # Number of visits }) # Simulate conversions (more likely for treatment) data['converted'] = ( np.random.binomial(1, 0.10, n_users) | # Base conversion rate (data['variant'] == 'treatment') & np.random.binomial(1, 0.03, n_users) # +3% for treatment ).astype(int) # Simulate revenue (higher for converters and treatment) data['revenue'] = 0.0 converters = data['converted'] == 1 data.loc[converters, 'revenue'] = np.random.gamma(shape=2, scale=25, size=converters.sum()) # Treatment group gets slightly higher revenue treatment_converters = (data['variant'] == 'treatment') & converters data.loc[treatment_converters, 'revenue'] *= 1.15 print(f"Dataset shape: {data.shape}") print(f"\nFirst few rows:") data.head(10)

Dataset shape: (2000, 5)

First few rows:

Out[4]:

	user_id	variant	visits	converted	revenue
0	0	control	7	1	90.366149
1	1	treatment	14	0	0.000000
2	2	control	13	0	0.000000
3	3	control	7	0	0.000000
4	4	control	16	0	0.000000
5	5	treatment	7	0	0.000000
6	6	control	15	0	0.000000
7	7	control	12	0	0.000000
8	8	control	16	0	0.000000
9	9	treatment	8	0	0.000000

2. Define Analysis Plan¶

Now let's define our analysis plan with multiple metrics:

• conversions: Simple metric counting total conversions
• conversion_rate: Ratio metric (conversions / visits)
• revenue: Simple metric for total revenue

In [5]:

from cluster_experiments import (
    AnalysisPlan, SimpleMetric, RatioMetric, 
    Variant, HypothesisTest
)

# Define metrics by type
simple_metrics = {
    "conversions": "converted",  # alias: column_name
    "revenue": "revenue"
}

ratio_metrics = {
    "conversion_rate": {
        "numerator": "converted",
        "denominator": "visits"
    }
}

# Define variants
variants = [
    Variant("control", is_control=True),
    Variant("treatment", is_control=False)
]

# Build hypothesis tests
hypothesis_tests = []

# Ratio metrics: use delta method
for alias, config in ratio_metrics.items():
    metric = RatioMetric(
        alias=alias,
        numerator_name=config["numerator"],
        denominator_name=config["denominator"]
    )
    hypothesis_tests.append(
        HypothesisTest(
            metric=metric,
            analysis_type="delta",
            analysis_config={
                "scale_col": metric.denominator_name,
                "cluster_cols": ["user_id"]
            }
        )
    )

# Simple metrics: use OLS
for alias, column_name in simple_metrics.items():
    metric = SimpleMetric(alias=alias, name=column_name)
    hypothesis_tests.append(
        HypothesisTest(
            metric=metric,
            analysis_type="ols"
        )
    )

# Create analysis plan
analysis_plan = AnalysisPlan(
    tests=hypothesis_tests,
    variants=variants,
    variant_col='variant'
)

print("Analysis plan created successfully!")

from cluster_experiments import ( AnalysisPlan, SimpleMetric, RatioMetric, Variant, HypothesisTest ) # Define metrics by type simple_metrics = { "conversions": "converted", # alias: column_name "revenue": "revenue" } ratio_metrics = { "conversion_rate": { "numerator": "converted", "denominator": "visits" } } # Define variants variants = [ Variant("control", is_control=True), Variant("treatment", is_control=False) ] # Build hypothesis tests hypothesis_tests = [] # Ratio metrics: use delta method for alias, config in ratio_metrics.items(): metric = RatioMetric( alias=alias, numerator_name=config["numerator"], denominator_name=config["denominator"] ) hypothesis_tests.append( HypothesisTest( metric=metric, analysis_type="delta", analysis_config={ "scale_col": metric.denominator_name, "cluster_cols": ["user_id"] } ) ) # Simple metrics: use OLS for alias, column_name in simple_metrics.items(): metric = SimpleMetric(alias=alias, name=column_name) hypothesis_tests.append( HypothesisTest( metric=metric, analysis_type="ols" ) ) # Create analysis plan analysis_plan = AnalysisPlan( tests=hypothesis_tests, variants=variants, variant_col='variant' ) print("Analysis plan created successfully!")

Analysis plan created successfully!

3. Run Analysis¶

Let's run the analysis and generate a comprehensive scorecard.

In [6]:

# Run analysis
results = analysis_plan.analyze(data)

# View results as a dataframe
results_df = results.to_dataframe()
print("\n=== Experiment Results ===")
results_df

# Run analysis results = analysis_plan.analyze(data) # View results as a dataframe results_df = results.to_dataframe() print("\n=== Experiment Results ===") results_df

=== Experiment Results ===

Out[6]:

	metric_alias	control_variant_name	treatment_variant_name	control_variant_mean	treatment_variant_mean	analysis_type	ate	ate_ci_lower	ate_ci_upper	p_value	std_error	dimension_name	dimension_value	alpha
0	conversion_rate	control	treatment	0.009972	0.011912	delta	0.001940	-0.000825	0.004706	0.169006	0.001411	__total_dimension	total	0.05
1	conversions	control	treatment	0.100394	0.117886	ols	0.017492	-0.009874	0.044859	0.210285	0.013963	__total_dimension	total	0.05
2	revenue	control	treatment	5.451515	7.359327	ols	1.907812	-0.130488	3.946112	0.066581	1.039968	__total_dimension	total	0.05

Summary¶

This example demonstrated:

1. ✅ Data Simulation: Creating realistic experiment data
2. ✅ Multiple Metric Types: Analyzing both simple and ratio metrics
3. ✅ Easy Configuration: Using dictionary-based analysis plan setup
4. ✅ Comprehensive Results: Getting treatment effects, confidence intervals, and p-values

Next Steps¶

• Try the CUPAC example to learn about variance reduction
• Explore cluster randomization for handling correlated units
• Learn about switchback experiments for time-based designs