Free Trial Screener

  • Udacity Experiment

Unit of diversion

  • Cookie

Experiment Design

Metric Choice

Invariant Metrics

  • Number of cookies
  • Number of clicks
  • Click through probability

Evaluation Metrics

  • Gross Conversion
  • Net Conversion

Initial Hypothesis

  • “..this might set clearer expectations for students upfront, thus reducing the number of frustrated students who left the free trial because they didn't have enough time”
  • ”..without significantly reducing the number of students to continue past the free trial and eventually complete the course. “

Measuring Standard Deviation

  • Gross Conversion: 0.02
  • Net Conversion 0.0156

Expect analytical variance match empirical variance because unit of analysis and unit of diversion is same.

Sizing

Number of Samples vs. Power

  • Gross Conversion. Baseline: 0.20625 dmin: 0.01 = 25.839 cookies who clicks.
  • Net Conversion. Baseline: 0.1093125 dmin: 0.0075 = 27,411 cookies who clicks.
  • Not using Bonferroni correction.
  • Using alpha = 0.05 and beta 0.2

The pageviews needed then will be: 685275 days.

Duration vs. Exposure

  • Fraction: 0.8 (Low risk)
  • Duration: 22 days (40000 pageviews/day)

In [ ]: