There is only one test

Copyright 2018 Allen Downey

License: http://creativecommons.org/licenses/by/4.0/


In [1]:
# Configure Jupyter so figures appear in the notebook
%matplotlib inline

# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

import numpy as np
import matplotlib.pyplot as plt

The tumbleweed problem

Here's an example that uses the "There is only one test" framework to solve a problem from the 2019 AP® Statistics Exam

Tumbleweed, commonly found in the western United States, is the dried structure of certain plants that are blown by the wind. Kochia, a type of plant that turns into tumbleweed at the end of the summer, is a problem for farmers because it takes nutrients away from soil that would otherwise go to more beneficial plants. Scientists are concerned that kochia plants are becoming resistant to the most commonly used herbicide, glyphosate.

In 2014, 19.7 percent of 61 randomly selected kochia plants were resistant to glyphosate. In 2017, 38.5 percent of 52 randomly selected kochia plants were resistant to glyphosate. Do the data provide convincing statistical evidence, at the level of $\alpha = 0.05$, that there has been an increase in the proportion of all kochia plants that are resistant to glyphosate?

Because the problem reports the data in terms of percentages, we have to work backwards to get the actual number of plants in each group that were resistant:


In [2]:
0.197 * 61


Out[2]:
12.017000000000001

In [3]:
0.385 * 52


Out[3]:
20.02

It looks like there were 12 resistant plants in the first group and 20 in the second group.


In [4]:
k1 = 12
k2 = 20
k = k1 + k2


Out[4]:
32

And here are the sample sizes.


In [5]:
n1 = 61
n2 = 52
n = n1 + n2


Out[5]:
113

The first step toward hypothesis testing is to define a test statistic that measures the size of the effect. In this case, the hypothetical effect is an increase in the fraction of resistant plants, that is, a difference in proportions.

I'll write a function that takes the data as a parameter and computes the difference in proportions.


In [6]:
def test_stat(data):
    """Compute the test statistic."""
    k1, n1, k2, n2 = data
    p1 = k1/n1
    p2 = k2/n2
    return p2 - p1

I pass the data to the function as a tuple of 4 numbers.


In [7]:
data = k1, n1, k2, n2


Out[7]:
(12, 61, 20, 52)

Now we can compute the observed difference in proportions (and check it against the numbers in the problem).


In [8]:
actual = test_stat(data)


Out[8]:
0.1878940731399748

In [9]:
0.385 - 0.197


Out[9]:
0.188

Now we need a null hypothesis. The scientists think there might be an increase in the fraction of resistant plants, so the null hypothesis is that there is no increase.

Under that assumption, the two groups have the same proportion of resistant plants, which we can estimate by pooling the data and computing the proportion among all of the observed plants.


In [10]:
p = k / n


Out[10]:
0.2831858407079646

Now, the fundamental question in hypothesis testing is this: under the null hypothesis, what would be the chance of seeing an apparent effect as big as actual?

To answer that question, we simulate the null hypothesis and generate "fake data", then compute the test statistic of the fake data.

I'll use the following function, which simulates flipping n coins, each with probability p of landing heads, and returns the total number of heads.


In [11]:
def flip(n, p):
    """Flip n coins with probability p.
    
    n: number of coins
    p: probability of heads
    
    return: total number of heads
    """
    return np.sum(np.random.random(n) < p)

As an example, I'll simulate the two groups and compute the number of resistant plants in each group.


In [12]:
flip(n1, p)


Out[12]:
24

In [13]:
flip(n2, p)


Out[13]:
17

Now we can wrap that in a function that returns the results in the same format as the original data, a tuple of 4 numbers.


In [14]:
def run_model(data):
    """Simulate the null hypothesis."""
    k1, n1, k2, n2 = data
    p = (k1 + k2) / (n1 + n2)
    k1 = flip(n1, p)
    k2 = flip(n2, p)
    return k1, n1, k2, n2

Returning the results in that format makes it possible to pass the "fake data" directly to test_stat.


In [15]:
test_stat(run_model(data))


Out[15]:
0.0012610340479193016

That's the result of one simulated experiment.

Now we run the experiment 1000 times and collect the results.


In [16]:
test_stat_dist = np.array([test_stat(run_model(data)) 
                           for i in range(1000)])


Out[16]:
array([-0.1056116 ,  0.00693569, -0.00945776, -0.11349306, -0.01229508,
        0.10025221,  0.07534678, -0.02301387,  0.00693569, -0.15983607,
        0.00126103, -0.16551072,  0.02332913, -0.07786885, -0.15479193,
        0.00189155, -0.04224464, -0.07786885,  0.10813367, -0.02868852,
        0.20428752,  0.10025221,  0.0759773 ,  0.03972257, -0.04791929,
       -0.0037831 , -0.08858764, -0.00662043, -0.08921816,  0.01544767,
       -0.06147541,  0.07534678, -0.09993695,  0.11380832, -0.04224464,
       -0.12988651, -0.1056116 , -0.00094578,  0.1223203 , -0.18757881,
        0.03184111, -0.02301387, -0.04508197,  0.02332913, -0.1056116 ,
        0.05390921,  0.02900378, -0.08354351,  0.00409836, -0.06147541,
        0.11948298, -0.0501261 ,  0.09741488, -0.03940731,  0.08102144,
       -0.00662043,  0.11380832, -0.08575032, -0.08070618, -0.11916772,
       -0.01450189, -0.0258512 , -0.0721942 ,  0.110971  ,  0.10025221,
       -0.04791929,  0.02332913, -0.04224464,  0.05958386,  0.00977301,
       -0.08858764,  0.03972257,  0.02332913, -0.03089533, -0.11633039,
       -0.03940731,  0.00409836, -0.07786885, -0.02017654,  0.00693569,
        0.02332913,  0.08669609, -0.12988651, -0.1406053 , -0.12137453,
        0.04823455,  0.06179067,  0.11664565,  0.02332913,  0.06746532,
        0.07030265,  0.07818411,  0.1223203 ,  0.15226986, -0.05296343,
        0.16078184, -0.0372005 ,  0.04823455,  0.1223203 , -0.01796974,
        0.02616646, -0.09142497, -0.1056116 , -0.05580076,  0.01544767,
        0.0425599 , -0.09993695,  0.01765448, -0.04508197,  0.03184111,
        0.1223203 , -0.03373266,  0.02332913, -0.11916772,  0.02112232,
       -0.11349306,  0.07030265,  0.08102144,  0.06746532,  0.07250946,
       -0.15983607, -0.04791929,  0.00977301,  0.06462799, -0.12704918,
       -0.00945776,  0.01544767, -0.06431274,  0.14659521,  0.07250946,
       -0.04224464, -0.08070618,  0.06462799,  0.04539723, -0.04508197,
        0.08606557,  0.00977301,  0.05674653, -0.03940731, -0.05296343,
       -0.0721942 ,  0.08669609, -0.05580076,  0.20996217,  0.00189155,
       -0.03436318, -0.08070618,  0.07534678, -0.29728878,  0.05107188,
        0.02332913, -0.01513241, -0.15479193,  0.02112232,  0.07250946,
       -0.11349306,  0.02900378,  0.18505675,  0.03688525,  0.02900378,
        0.08385876, -0.0942623 ,  0.01261034,  0.05390921, -0.07503153,
        0.09741488, -0.11349306,  0.0425599 , -0.08070618,  0.03751576,
        0.02616646, -0.01229508,  0.19861286, -0.02805801,  0.05107188,
        0.12452711,  0.10592686, -0.10277427, -0.07503153, -0.03656999,
       -0.06431274, -0.02805801,  0.08102144, -0.11916772, -0.0258512 ,
       -0.03940731,  0.25409836, -0.01450189,  0.20208071,  0.04823455,
        0.09457755,  0.12515763,  0.06462799,  0.07250946, -0.03656999,
       -0.01733922, -0.00094578, -0.08070618,  0.04539723,  0.07313997,
       -0.03089533,  0.05107188, -0.03152585,  0.02900378,  0.00693569,
        0.09741488,  0.02112232,  0.00977301,  0.01261034, -0.11412358,
        0.22068096, -0.01229508, -0.11349306, -0.00094578,  0.03467844,
        0.23139975, -0.05075662, -0.0037831 , -0.12988651, -0.0942623 ,
        0.10025221,  0.09741488,  0.10876419, -0.07503153,  0.08953342,
       -0.02868852,  0.05327869,  0.07534678,  0.05107188,  0.0425599 ,
        0.02900378, -0.10277427, -0.03152585,  0.00977301,  0.07250946,
       -0.04224464,  0.06179067,  0.03467844, -0.10844893,  0.03467844,
        0.17150063,  0.00693569,  0.03972257,  0.13871375,  0.00977301,
       -0.13776797,  0.00977301, -0.12421185, -0.00094578, -0.06147541,
       -0.09142497, -0.01229508, -0.11633039, -0.14407314,  0.03184111,
       -0.03152585,  0.12168979,  0.04823455,  0.09457755,  0.03972257,
       -0.0721942 ,  0.05044136,  0.09174023,  0.02616646,  0.09457755,
        0.10245902, -0.10844893,  0.00126103,  0.14218159,  0.00409836,
       -0.13556116, -0.03656999, -0.04003783,  0.10876419,  0.04035309,
        0.110971  , -0.16834805, -0.05580076,  0.0889029 ,  0.13587642,
        0.04539723, -0.00945776, -0.0258512 ,  0.10592686, -0.05580076,
       -0.1406053 ,  0.07818411, -0.0942623 , -0.08070618, -0.0721942 ,
       -0.08070618,  0.06462799,  0.06179067, -0.03089533, -0.01513241,
        0.15226986, -0.00945776,  0.00977301,  0.10876419,  0.10592686,
        0.15510719, -0.02080706, -0.00157629, -0.08354351, -0.09142497,
       -0.08070618, -0.02017654, -0.06651955,  0.04539723,  0.03467844,
       -0.03940731, -0.11633039,  0.10025221,  0.1223203 , -0.10277427,
        0.01481715,  0.00977301,  0.05895334, -0.11412358,  0.04035309,
       -0.04508197, -0.19546028,  0.00126103, -0.08070618,  0.01828499,
        0.28121059,  0.01828499,  0.02616646, -0.08070618,  0.07534678,
        0.02900378,  0.05107188,  0.01544767,  0.06746532,  0.06746532,
       -0.04508197,  0.06179067,  0.06462799,  0.02616646, -0.01513241,
       -0.01733922,  0.03184111,  0.00693569, -0.02868852,  0.06179067,
        0.02616646,  0.06179067, -0.06431274,  0.07534678, -0.14911728,
       -0.00094578, -0.15762926, -0.05580076, -0.11349306,  0.03530895,
       -0.05359395, -0.15479193, -0.04224464,  0.00977301, -0.11916772,
        0.0889029 ,  0.01544767, -0.09142497, -0.00441362, -0.12704918,
        0.05895334,  0.00693569,  0.1223203 , -0.00945776, -0.09993695,
       -0.04224464,  0.01828499, -0.11349306,  0.11664565, -0.08070618,
        0.1223203 , -0.00945776, -0.02017654, -0.16551072, -0.11633039,
       -0.10498108,  0.1443884 ,  0.00977301, -0.14691047, -0.03152585,
        0.13240858, -0.14911728, -0.07503153, -0.07503153,  0.16078184,
        0.02616646,  0.03184111,  0.08669609, -0.00662043,  0.09174023,
       -0.00945776, -0.09709962, -0.07503153, -0.11916772, -0.00945776,
       -0.1056116 ,  0.04539723,  0.06746532,  0.08669609, -0.12704918,
        0.13303909, -0.06998739,  0.10025221, -0.08070618, -0.00945776,
        0.00126103, -0.03940731,  0.04823455, -0.00945776,  0.07030265,
        0.02616646, -0.00725095, -0.12988651,  0.03467844, -0.08575032,
       -0.11633039,  0.01828499,  0.09741488, -0.03940731, -0.04508197,
        0.03404792,  0.00126103,  0.00409836,  0.07818411,  0.04539723,
       -0.11349306, -0.13556116,  0.14722573, -0.03940731, -0.02868852,
       -0.05359395, -0.03940731,  0.10592686, -0.03940731,  0.12168979,
        0.07534678,  0.01261034,  0.04539723, -0.01513241, -0.07786885,
       -0.06651955, -0.20901639, -0.1056116 ,  0.06179067, -0.09205549,
       -0.11349306,  0.13303909, -0.08070618,  0.10308953, -0.00094578,
       -0.15479193, -0.12988651, -0.09993695, -0.01513241, -0.05296343,
       -0.03656999, -0.04224464, -0.05863808, -0.02017654,  0.0425599 ,
        0.04035309, -0.12704918, -0.00662043,  0.03688525, -0.06431274,
        0.03751576, -0.0258512 , -0.00441362,  0.00693569,  0.19356873,
       -0.05863808, -0.11065574, -0.06147541,  0.10025221, -0.01796974,
        0.08102144,  0.02900378, -0.02017654, -0.01513241,  0.07030265,
        0.00126103, -0.02301387,  0.0425599 ,  0.14155107,  0.01828499,
        0.09457755, -0.04224464,  0.00693569, -0.17622951,  0.11380832,
        0.08102144,  0.10308953,  0.04539723,  0.02332913,  0.02900378,
       -0.03940731,  0.06746532, -0.08921816,  0.06462799, -0.0721942 ,
        0.08385876,  0.17213115, -0.0721942 ,  0.01197982, -0.0258512 ,
       -0.0964691 ,  0.11664565,  0.0425599 , -0.04224464, -0.12421185,
        0.23203026, -0.04508197,  0.12799496,  0.01261034,  0.03467844,
       -0.12421185,  0.06179067, -0.04508197,  0.1686633 , -0.01229508,
       -0.07786885,  0.11380832, -0.0721942 ,  0.07818411,  0.02616646,
       -0.08354351, -0.08354351, -0.03656999,  0.03688525, -0.07282472,
       -0.01513241, -0.11349306, -0.00662043, -0.05296343,  0.07818411,
        0.05107188, -0.03940731, -0.01229508,  0.02332913, -0.00945776,
        0.14155107, -0.00945776, -0.0037831 ,  0.00409836,  0.22351828,
       -0.01166456, -0.12704918,  0.02900378, -0.0258512 ,  0.07030265,
        0.02616646, -0.19262295,  0.23486759, -0.00945776, -0.0721942 ,
       -0.04508197,  0.00472888,  0.03972257, -0.01796974, -0.13272383,
        0.06746532, -0.00094578, -0.18978562,  0.04823455, -0.03436318,
       -0.00945776,  0.06179067, -0.16551072, -0.00441362,  0.04476671,
       -0.06368222,  0.16298865,  0.17938209,  0.00977301,  0.02332913,
        0.1443884 , -0.04508197,  0.01261034,  0.01261034,  0.06179067,
       -0.05359395,  0.04823455, -0.11633039, -0.0258512 , -0.21469105,
       -0.08858764,  0.02332913,  0.00693569, -0.15983607, -0.05359395,
       -0.01229508,  0.0075662 ,  0.03972257,  0.09741488, -0.05296343,
       -0.01229508, -0.02301387, -0.01229508, -0.07786885,  0.0889029 ,
       -0.13556116, -0.03089533, -0.12200504, -0.1982976 , -0.04508197,
       -0.07786885, -0.06368222,  0.05895334, -0.01796974, -0.06147541,
        0.19577554,  0.1223203 ,  0.07818411, -0.00441362, -0.04508197,
       -0.05580076, -0.0258512 , -0.00945776,  0.08102144, -0.06147541,
       -0.04791929, -0.11916772, -0.02301387, -0.04791929,  0.00693569,
        0.01261034,  0.04539723,  0.04539723,  0.11948298,  0.08669609,
        0.10876419, -0.0258512 , -0.0037831 , -0.00157629, -0.0501261 ,
        0.11664565,  0.16078184,  0.08102144,  0.00472888, -0.07503153,
        0.02900378,  0.09237074,  0.11664565,  0.03751576,  0.0425599 ,
       -0.16551072,  0.10813367,  0.10308953, -0.07786885, -0.16834805,
       -0.0258512 ,  0.00693569,  0.01765448, -0.12704918, -0.02301387,
        0.08102144,  0.12168979, -0.04791929, -0.09709962,  0.04319042,
        0.10025221, -0.20680958,  0.09174023, -0.07786885, -0.02301387,
        0.03688525,  0.03751576, -0.16834805, -0.02868852,  0.11664565,
       -0.0372005 ,  0.09741488, -0.05296343, -0.15479193, -0.08921816,
       -0.12421185,  0.16078184, -0.1056116 , -0.02868852,  0.11664565,
       -0.07503153,  0.02395965, -0.00662043,  0.08102144, -0.19325347,
        0.06179067, -0.04224464,  0.00693569, -0.09142497, -0.08291299,
        0.03184111, -0.06715006, -0.04791929,  0.15510719,  0.09237074,
       -0.0037831 , -0.07503153, -0.03436318,  0.01544767,  0.04823455,
        0.15510719, -0.00662043, -0.05075662, -0.00662043,  0.04539723,
        0.0425599 , -0.05296343,  0.09741488,  0.14155107,  0.01261034,
        0.00126103, -0.07503153,  0.05611602,  0.06462799, -0.06431274,
       -0.08354351, -0.03940731, -0.05863808, -0.04508197,  0.01765448,
        0.08448928, -0.03152585,  0.02616646, -0.10781841, -0.02868852,
        0.04319042,  0.01481715,  0.0075662 , -0.0942623 , -0.14911728,
        0.02616646,  0.04035309, -0.04224464, -0.05359395, -0.06651955,
       -0.0721942 , -0.09142497, -0.04728878,  0.13303909, -0.01796974,
        0.06462799, -0.02080706, -0.1519546 , -0.04508197,  0.01828499,
       -0.04508197,  0.08953342, -0.04224464,  0.10025221,  0.18505675,
        0.09741488,  0.05107188,  0.05044136,  0.00977301, -0.11349306,
        0.06179067,  0.07534678, -0.02017654, -0.04224464,  0.09741488,
       -0.04224464, -0.07786885, -0.11916772, -0.03152585,  0.1686633 ,
       -0.1740227 , -0.09709962, -0.08354351,  0.07313997,  0.11664565,
       -0.07786885, -0.13556116, -0.01513241,  0.02900378,  0.00977301,
       -0.04791929,  0.04823455, -0.0721942 ,  0.01765448,  0.09237074,
       -0.00094578,  0.08669609,  0.04823455, -0.07786885, -0.00662043,
       -0.12704918,  0.05327869,  0.16361917, -0.03940731,  0.11948298,
        0.05327869,  0.04539723,  0.02900378, -0.2203657 ,  0.05895334,
       -0.02017654, -0.01229508,  0.13303909, -0.02080706,  0.14092055,
       -0.12704918,  0.07534678,  0.05390921,  0.02900378, -0.21752837,
        0.18221942, -0.05296343, -0.04791929,  0.08385876,  0.04539723,
       -0.01513241, -0.14911728,  0.11948298, -0.08638083, -0.01796974,
       -0.02301387,  0.05611602,  0.19640605,  0.01261034, -0.13556116,
        0.18505675,  0.09741488,  0.02332913,  0.00977301, -0.0721942 ,
        0.02616646,  0.09457755,  0.03972257,  0.07030265,  0.0204918 ,
        0.09741488,  0.09174023, -0.03436318, -0.06715006, -0.09142497,
        0.13871375,  0.08953342, -0.00882724, -0.00157629, -0.01796974,
       -0.05075662,  0.08165195, -0.09993695,  0.22351828,  0.11664565,
       -0.00945776, -0.06431274, -0.06147541, -0.0372005 ,  0.07313997,
       -0.06431274,  0.15794451,  0.00977301, -0.13839849, -0.11065574,
       -0.0721942 ,  0.08385876, -0.05580076, -0.22320303,  0.08385876,
       -0.18757881,  0.03688525,  0.12452711, -0.11065574,  0.07030265,
        0.10308953, -0.15762926,  0.01261034, -0.01229508,  0.00126103,
       -0.00945776,  0.01544767,  0.05044136, -0.05296343,  0.06967213,
        0.10813367, -0.00094578, -0.02017654,  0.09457755,  0.02332913,
       -0.00094578,  0.00977301, -0.13839849, -0.10781841,  0.10025221,
        0.10592686,  0.03184111,  0.05107188, -0.0258512 ,  0.01765448,
        0.13020177, -0.05359395, -0.03940731, -0.17906683, -0.12421185,
        0.05390921,  0.0425599 , -0.08070618, -0.01796974, -0.05863808,
        0.02332913,  0.07313997, -0.11065574, -0.11349306, -0.07786885,
        0.10025221, -0.01229508, -0.2203657 ,  0.0204918 ,  0.14659521,
       -0.08858764, -0.04508197, -0.15762926,  0.06746532, -0.08575032,
        0.02616646,  0.02900378,  0.08669609, -0.21469105, -0.0501261 ,
       -0.02301387,  0.07250946, -0.04003783,  0.03467844,  0.06179067,
       -0.06147541, -0.00315259, -0.1056116 ,  0.00472888, -0.03436318,
       -0.02364439,  0.08102144,  0.03467844, -0.06715006,  0.00189155,
        0.13587642, -0.07503153,  0.08385876,  0.00126103,  0.08102144,
       -0.12421185, -0.01513241, -0.10498108, -0.07503153, -0.18190416,
        0.05390921, -0.03940731, -0.06431274, -0.07503153, -0.0037831 ,
        0.03972257,  0.01261034,  0.07250946, -0.0037831 ,  0.07818411,
        0.12452711,  0.17150063, -0.02301387, -0.02868852, -0.05296343,
        0.03184111, -0.02868852, -0.03940731,  0.00126103, -0.09993695,
       -0.01229508, -0.00094578, -0.06431274, -0.18474149,  0.06179067,
       -0.14911728,  0.05390921,  0.13587642, -0.00945776,  0.10025221,
        0.01261034, -0.08858764, -0.08575032,  0.09174023, -0.11633039,
        0.08385876,  0.03184111, -0.06715006, -0.01229508, -0.0258512 ,
        0.02332913, -0.00945776, -0.03656999,  0.07818411, -0.01229508,
       -0.0258512 ,  0.01765448,  0.02900378,  0.00977301,  0.00126103])

The result is the "sampling distribution of the test statistic under the null hypothesis".

The mean of this distribution is close to zero, which is not suprising because it is based on the assumption that there is actually no difference between the groups.


In [17]:
np.mean(test_stat_dist)


Out[17]:
-0.0010378310214375767

Here's what the distribution of test statistics looks like.

The vertical gray line shows the observed effect size, actual.


In [18]:
plt.hist(test_stat_dist)
plt.axvline(actual, color='gray')
    
plt.xlabel('Difference in proportions')
plt.ylabel('Number of simulations')
plt.title('Sampling distribution of the test statistic under H0');


Now we can compute the probability that the test statistic, under the null hypothesis, exceeds the observed difference.

This probability is the "p-value".


In [19]:
p_value = np.mean(test_stat_dist >= actual)


Out[19]:
0.015

The estimated p-value is less than the specified threshold, $\alpha = 0.05$, so the observed difference is considered statistically significant.

Different test statistics

When I present the "There is only one test" framework, many of the questions I get are about choosing the test statistic. In many cases, it seems like an arbitrary choice.

And in some ways it is, although rather than "arbitrary", I would say it is a modeling decision informed by the context. In this example, the researchers think there might be an increase in the proportion of resistant plants, so I chose a test statistic that measures an increase.

If the researchers expected a change in resistance without specifying an increase or decrease, I might use the absolute value of the difference as a test statistic, because the absolute value measures magnitude without regard to sign.

Using the difference in proportions was also a choice; another possibility is to use the ratio of proportions. In the context of epidemiology, it is common to use risk ratios to compare proportions.

One benefit of the computational framework is that it is easy to try a different test statistic (or a different model of the null hypothesis) and see what effect it has on the results.

Here's a function that computes the ratio of proportions.


In [20]:
def test_stat(data):
    k1, n1, k2, n2 = data
    p1 = k1/n1
    p2 = k2/n2
    return p2 / p1

Here's the observed risk ratio.


In [21]:
actual = test_stat(data)


Out[21]:
1.9551282051282053

Now we can run the simulated experiments with this test statistic, and compute the p-value.


In [22]:
test_stat_dist = np.array([test_stat(run_model(data)) 
                       for i in range(1000)])

p_value = np.mean(test_stat_dist >= actual)


Out[22]:
0.016

The two test statistics yield pretty much the same p-values. That's good; it means that the interpretation of the results is not sensitive to our modeling choices.


In [ ]: