Multiple treatments

This notebook shows how to use the multivariate module. The idea is to use several treatments in the splitter and only one of them is used to run the hypothesis test.

In [1]:

import numpy as np
import pandas as pd
from cluster_experiments import PowerAnalysis

import numpy as np import pandas as pd from cluster_experiments import PowerAnalysis

In [2]:

# Create fake data
N = 1_000
df = pd.DataFrame(
    {
        "target": np.random.normal(0, 1, size=N),
    }
)

# Create fake data N = 1_000 df = pd.DataFrame( { "target": np.random.normal(0, 1, size=N), } )

In [3]:

# Run power analysis using 3 variants
config_abc = {
    "analysis": "ols_non_clustered",
    "perturbator": "constant",
    "splitter": "non_clustered",
    "treatments": ["A", "B", "C"],
    "control": "A",
    "treatment": "B",
    "n_simulations": 50,
}

power_abc = PowerAnalysis.from_dict(config_abc)
power_abc.power_analysis(df, average_effect=0.1)

# Run power analysis using 3 variants config_abc = { "analysis": "ols_non_clustered", "perturbator": "constant", "splitter": "non_clustered", "treatments": ["A", "B", "C"], "control": "A", "treatment": "B", "n_simulations": 50, } power_abc = PowerAnalysis.from_dict(config_abc) power_abc.power_analysis(df, average_effect=0.1)

Out[3]:

0.18

In [4]:

# Run power analysis using 2 variants
config_ab = {
    "analysis": "ols_non_clustered",
    "perturbator": "constant",
    "splitter": "non_clustered",
    "treatments": ["A", "B"],
    "control": "A",
    "treatment": "B",
    "n_simulations": 50,
}
power_ab = PowerAnalysis.from_dict(config_ab)
power_ab.power_analysis(df, average_effect=0.1)

# Run power analysis using 2 variants config_ab = { "analysis": "ols_non_clustered", "perturbator": "constant", "splitter": "non_clustered", "treatments": ["A", "B"], "control": "A", "treatment": "B", "n_simulations": 50, } power_ab = PowerAnalysis.from_dict(config_ab) power_ab.power_analysis(df, average_effect=0.1)

Out[4]:

0.28

The power of the AB test is higher than the ABC test, which makes sense.