Ebisu howto

A quick introduction to using the library to schedule spaced-repetition quizzes in a principled, probabilistically-grounded, Bayesian manner.

See https://fasiha.github.io/ebisu/ for details!



In [1]:

    
import ebisu

defaultModel = (4., 4., 24.) # alpha, beta, and half-life in hours

Ebisu—this is what we’re here to learn about!

Ebisu is a library that’s expected to be embedded inside quiz apps, to help schedule quizzes intelligently. It uses Bayesian statistics to let the app predict what the recall probability is for any fact that the student has learned, and to update that prediction based onthe results of a quiz.

Ebisu uses three numbers to describe its belief about the time-evolution of each fact’s recall probability. Its API consumes them as a 3-tuple, and they are:

the first we call “alpha” and must be ≥ 2 (well, technically, ≥1 is the raw minimum but unless you’re a professor of statistics, keep it more than two);
the second is “beta” and also must be ≥ 2. These two numbers encode our belief about the distribution of recall probabilities at
the third element, which here is a half-life. This has units of time, and for this example, we’ll assume it’s in hours. It can be any positive float, but we choose the nice round number of 24 hours.

For the nerds: alpha and beta parameterize a Beta distribution to describe our prior belief of the recall probability one half-life (one day) after a fact’s most recent quiz.

For the rest of us: these three numbers mean we expect the recall probability for a newly-learned fact to be 50% after one day, but allow uncertainty: the recall probability after a day is “around” 42% to 58% (±1 standard deviation).

Now. Let’s create a mock database of facts. Say a student has learned two facts, one on the 19th at 2200 hours and another the next morning at 0900 hours.



In [2]:

    
from datetime import datetime, timedelta
date0 = datetime(2017, 4, 19, 22, 0, 0)

database = [dict(factID=1, model=defaultModel, lastTest=date0),
            dict(factID=2, model=defaultModel, lastTest=date0 + timedelta(hours=11))]

After learning the second fact, at 0900, what does Ebisu expect each fact’s probability of recall to be, for each of the facts?



In [3]:

    
oneHour = timedelta(hours=1)

now = date0 + timedelta(hours=11.1)
print("On {},".format(now))
for row in database:
    recall = ebisu.predictRecall(row['model'],
                                 (now - row['lastTest']) / oneHour,
                                 exact=True)
    print("Fact #{} probability of recall: {:0.1f}%".format(row['factID'], recall * 100))









    



On 2017-04-20 09:06:00,
Fact #1 probability of recall: 71.5%
Fact #2 probability of recall: 99.7%

Both facts are expected to still be firmly in memory—especially the second one since it was just learned! So the quiz app doesn’t ask the student to review anything yet—though if she wanted to, the quiz app would pick the fact most in danger of being forgotten.

Note how we used ebisu.predictRecall, which accepts

the current model, and
the time elapsed since this fact’s last quiz,

and returns a float.

…

Now a few hours have elapsed. It’s just past midnight on the 21st and the student opens the quiz app.



In [4]:

    
now = date0 + timedelta(hours=26.5)
print("On {},".format(now))
for row in database:
    recall = ebisu.predictRecall(row['model'],
                                 (now - row['lastTest']) / oneHour,
                                 exact=True)
    print("Fact #{} probability of recall: {:0.1f}%".format(row['factID'], recall * 100))









    



On 2017-04-21 00:30:00,
Fact #1 probability of recall: 46.8%
Fact #2 probability of recall: 63.0%

Suppose the quiz app has been configured to quiz the student if the expected recall probability drops below 50%—which it did for fact 1! The app shows the flashcard once, analyzes the user's response, and sets the result of the quiz to 1 if passed and 0 if failed. It calls Ebisu to update the model, giving it this result as well as the total number of times it showed this flashcard (one time—Ebisu can support more advanced cases where an app reviews the same flashcard multiple times in a single review session, but let's keep it simple for now).



In [5]:

    
row = database[0] # review FIRST question

result = 1 # success!
total = 1 # number of times this flashcard was shown (fixed)
newModel = ebisu.updateRecall(row['model'],
                              result,
                              total,
                              (now - row['lastTest']) / oneHour)
print('New model for fact #1:', newModel)
row['model'] = newModel
row['lastTest'] = now









    



New model for fact #1: (5.104166666666623, 3.999999999999968, 24.0)

Observe how ebisu.updateRecall takes

the current model,
the quiz result, and
the time elapsed since the last quiz,

and returns a new model (the new 3-tuple of “alpha”, “beta” and time). We put the new model and the current timestamp into the database.

Now. Suppose the student asks to review another fact—fact 2. It was learned just earlier that morning, and its recall probability is expected to be around 63%, but suppose the student fails this quiz, as sometimes happens.



In [6]:

    
row = database[1] # review SECOND question

result = 0
newModel = ebisu.updateRecall(row['model'],
                              result,
                              total,
                              (now - row['lastTest']) / oneHour)
print('New model for fact #2:', newModel)
row['model'] = newModel
row['lastTest'] = now









    



New model for fact #2: (3.897259022777383, 5.0345121960734005, 24.0)

The new parameters for this fact differ from the previous one because (1) the student failed this quiz while she passed the other, (2) different amounts of time had elapsed since the respective facts were last seen.

Ebisu provides a method to convert parameters to “expected half-life”. It is not an essential feature of the API and displaying it to the student will likely only distract them, but it is available:



In [7]:

    
for row in database:
    meanHalflife = ebisu.modelToPercentileDecay(row['model'])
    print("Fact #{} has half-life of ≈{:0.1f} hours".format(row['factID'], meanHalflife))









    



Fact #1 has half-life of ≈29.2 hours
Fact #2 has half-life of ≈19.8 hours

Note how the half-life (the time between quizzes for expected recall probability to drop to 50%) for the first question increased from 24 to 29 hours after the student got it right, while it decreased to 20 hours for the second when she got it wrong. Ebisu has incorporated the fact that the second fact had been learned not that long ago and should have been strong, and uses the surprising quiz result to strongly adjust its belief about its recall probability.

This short notebook shows the major functions in the Ebisu API:

ebisu.predictRecall to find out the expected recall probability for a fact right now, and
ebisu.updateRecall to update those expectations when a new quiz result is available.
ebisu.modelToPercentileDecay to find the time when the recall probability reaches a certain value.

Adanced topics

Speeding up `predictRecall`

Above, we used predictRecall with the exact=True keyword argument to have it return true probabilities. We can reduce runtime if we use the following:



In [8]:

    
# As above: a bit slow to get exact probabilities
%timeit ebisu.predictRecall(database[0]['model'], 100., exact=True)

# A bit faster alternative: get log-probabilities (this is the defa)
%timeit ebisu.predictRecall(database[0]['model'], 100., exact=False)









    



6.82 µs ± 84.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
4.91 µs ± 235 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Ebisu howto

Adanced topics

Speeding up predictRecall

Speeding up `predictRecall`