powered_effect is not calculated in the StudentsTTest

[jpzhangvincent]

I'm a bit confused why the powered_effect is not calculated in the StudentsTTest but it's provided in ZTest.

The above is the data frame which I passed into both

stat_res_df = confidence.ZTest(
    stats_df,
    numerator_column='conversions',
    numerator_sum_squares_column=None,
    denominator_column='total',
    categorical_group_columns='variant_id',
    correction_method='bonferroni')

and

stat_res_df = confidence.StudentsTTest(
    stats_df,
    numerator_column='conversions',
    numerator_sum_squares_column=None,
    denominator_column='total',
    categorical_group_columns='variant_id',
    correction_method='bonferroni')

but when I called stat_res_df.difference(level_1='control', level_2='treatment') I found the result from z-test provides the powered_effect column as below

but it's missing from the t-test result. Another question, why is the required_sample_size missing? Is there a way to also provide the sample size estimation in the result? Thanks!

[iampelle]

Since our sample sizes at Spotify are usually very large, it doesn't make a difference whether we use Z-tests or T-tests. Therefore we have mostly focused on the Z-test case and just not got around to implement everything for the other variants. It should be a simple thing to add though. The only difference should be in these lines, where we could use the corresponding t-distribution methods to get test statistics.

Time to make a first PR? 😉