Lesson 05 - Analyzing Results

Sanity Checks

• population and control population should be comparable
• invariant metrics should have remained invariant
• if sanity check fails
  • check with engineers whether there is a technical problem
  • retrospective analysis using the data to see whether there is something within the experiment that is causing this to happen

Single Metric Analysis

• Business decisions need to be made

Multiple Metric Analysis

• If checking 10 metrics then it is high probability that one of them says that it is significant but it's not. But that would be spurious
• Probability of false positive of any metric increases as number of metrics (n) increase
  • Solution
    • Assume independence
      • use alpha_overall = 1 - ((1 - alpha_individual) ^ n)
      • 95% confidence interval means alpha_individual is 0.05
    • Bonferron Correction
      • guaranteed to give alpha at least as small as specified
      • alpha_individual = alpha_overall / n