Exploring & Explaining churn prediction models

Churn prediction is the task of identifying users that are likely to stop using a service, product or website. In this tutorial, you will learn how to:

Explore & Evaluate predictions made by the model

  • Explore predictions made by this model to gain confidence in the model.
  • Understanding why the model made the predictions that it did make.
  • Make a churn report by segmenting users based on the their reasons for churn.
  • Evaluate the model and compare it with a baseline model.

Let's get started!


In [2]:
import graphlab as gl
import datetime
gl.canvas.set_target('ipynb') # make sure plots appear inline


A newer version of GraphLab Create (v2.0.1) is available! Your current version is v2.0.

You can use pip to upgrade the graphlab-create package. For more information see https://turi.com/products/create/upgrade.
[INFO] graphlab.cython.cy_server: GraphLab Create v2.0 started. Logging: /tmp/graphlab_server_1468133144.log
This commercial license of GraphLab Create is assigned to engr@turi.com.

Load previously saved data

In the previous notebooks, we had saved the data & models in a binary format. Let us try and load them back.


In [3]:
interactions_ts = gl.TimeSeries("data/user_activity_data.ts/")
users = gl.SFrame("data/users.sf/")
model = gl.load_model("data/churn_model.mdl")

In [4]:
(train, valid) = gl.churn_predictor.random_split(interactions_ts, user_id = 'CustomerID', fraction = 0.9, seed = 12)

In [5]:
churn_period_oct =  datetime.datetime(year = 2011, month = 10, day = 1)

Interactive view to explore the model


In [5]:
v = model.views.overview(train, churn_period_oct, valid, user_data=users)
v.show()


[WARNING] graphlab.toolkits.churn_predictor._churn_predictor: This feature is currently in beta. Please use with caution and not in mission-critical applications. For feedback or suggestions on this feature, please e-mail feedback@turi.com.
PROGRESS: Making a churn forecast for the time window:
PROGRESS: --------------------------------------------------
PROGRESS:  Start : 2011-10-01 00:00:00
PROGRESS:  End   : 2011-10-31 00:00:00
PROGRESS: --------------------------------------------------
PROGRESS: Grouping dataset by user.
PROGRESS: Resampling grouped observation_data by time-period 1 day, 0:00:00.
InvoiceNo is a categorical variable with too many different values (16841) and will be ignored.
StockCode is a categorical variable with too many different values (3649) and will be ignored.
Description is a categorical variable with too many different values (3845) and will be ignored.
PROGRESS: Generating features for boundary 2011-10-01 00:00:00.
PROGRESS: Joining user_data with aggregated features.
PROGRESS: Not enough data to make predictions for 657 user(s). 
PROGRESS: Making a churn forecast for the time window:
PROGRESS: --------------------------------------------------
PROGRESS:  Start : 2011-10-01 00:00:00
PROGRESS:  End   : 2011-10-31 00:00:00
PROGRESS: --------------------------------------------------
PROGRESS: Grouping dataset by user.
PROGRESS: Resampling grouped observation_data by time-period 1 day, 0:00:00.
InvoiceNo is a categorical variable with too many different values (16841) and will be ignored.
StockCode is a categorical variable with too many different values (3649) and will be ignored.
Description is a categorical variable with too many different values (3845) and will be ignored.
PROGRESS: Generating features for boundary 2011-10-01 00:00:00.
PROGRESS: Joining user_data with aggregated features.
PROGRESS: Not enough data to make predictions for 66 user(s). 
PROGRESS: Making a churn forecast for the time window:
PROGRESS: --------------------------------------------------
PROGRESS:  Start : 2011-10-01 00:00:00
PROGRESS:  End   : 2011-10-31 00:00:00
PROGRESS: --------------------------------------------------
PROGRESS: Grouping dataset by user.
PROGRESS: Resampling grouped observation_data by time-period 1 day, 0:00:00.
InvoiceNo is a categorical variable with too many different values (16841) and will be ignored.
StockCode is a categorical variable with too many different values (3649) and will be ignored.
Description is a categorical variable with too many different values (3845) and will be ignored.
PROGRESS: Generating features for boundary 2011-10-01 00:00:00.
PROGRESS: Joining user_data with aggregated features.
PROGRESS: Not enough data to make predictions for 66 user(s). 

What are the key features that impact churn?


In [6]:
importance = model.get_feature_importance()

In [7]:
print "What are the top 5 factors that impact predictions?"
print "----------------------------------------------------"
print '\n'.join(["%s. %s" % (i+1, x) for i,x in enumerate(importance['description'][0:5])])


What are the top 5 factors that impact predictions?
----------------------------------------------------
1. Days since most recent event
2. Sum of "Quantity" in the last 60 days
3. Sum of "Quantity" in the last 90 days
4. Days since the first event in the last 90 days
5. 90 day trend in the number of events

Segmenting groups of users with similar churn explanations


In [9]:
report = model.get_churn_report(interactions_ts, user_data=users)
report


PROGRESS: Making a churn forecast for the time window:
PROGRESS: --------------------------------------------------
PROGRESS:  Start : 2011-12-09 12:50:00
PROGRESS:  End   : 2012-01-08 12:50:00
PROGRESS: --------------------------------------------------
PROGRESS: Grouping dataset by user.
PROGRESS: Resampling grouped observation_data by time-period 1 day, 0:00:00.
InvoiceNo is a categorical variable with too many different values (22061) and will be ignored.
StockCode is a categorical variable with too many different values (4058) and will be ignored.
Description is a categorical variable with too many different values (4195) and will be ignored.
PROGRESS: Generating features for boundary 2011-12-09 12:50:00.
PROGRESS: Joining user_data with aggregated features.
Out[9]:
segment_id num_users num_users_percentage explanation avg_probability stdv_probability
0 633 14.5852534562 [No events in the last 90
days, Feature 'user_t ...
0.936356472366 0.0184749032653
1 465 10.7142857143 [No events in the last 90
days, Feature 'user_t ...
0.863619537123 0.0516703975442
2 391 9.00921658986 [No "Quantity" events in
the last 21 days, Sum of ...
0.560969559738 0.0684262171812
3 291 6.70506912442 [No events in the last 90
days, Feature 'user_t ...
0.942066940245 0.0277551105101
4 221 5.09216589862 [No events in the last 90
days, No events in the ...
0.640921664723 0.0258596294124
5 206 4.7465437788 [Sum of "Quantity" in the
last 90 days greater ...
0.921328827883 0.0375340152673
7 151 3.47926267281 [Sum of "Quantity" is
less than 109.50 each ...
0.526150263501 0.0907998464538
6 151 3.47926267281 [Less than 48.00 days
since most recent event, ...
0.146823709078 0.0918502770671
8 130 2.99539170507 [No events in the last 90
days, No days with an ...
0.757004905206 0.0345380212176
9 128 2.94930875576 [Sum of "Quantity" is
greater than (or equal ...
0.365221123851 0.169689745667
users
[12346, 12350, 12353,
12355, 12361, 12373, ...
[12363, 12365, 12410,
12414, 12418, 12422, ...
[12370, 12372, 12393,
12407, 12408, 12412, ...
[12367, 12375, 12403,
12430, 12442, 12454, ...
[12378, 12383, 12399,
12405, 12453, 12502, ...
[12374, 12390, 12420,
12425, 12427, 12435, ...
[12356, 12409, 15312,
13815, 15329, 13824, ...
[12362, 12437, 12471,
12474, 12476, 12490, ...
[12354, 12377, 12501,
16795, 15353, 15415, ...
[12381, 16779, 16790,
15351, 13854, 16814, ...
[79 rows x 7 columns]
Note: Only the head of the SFrame is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.


In [10]:
report['num_users'].show()


What does a segment look like?


In [11]:
segment = report[report['segment_id'] == '2'][0]

In [14]:
print ""
print "Segment 2"
print "---------------------------------------"
print 'Segment size      : %.2f %% of users' % segment["num_users_percentage"]
print 'Churn probability : %s' % segment["avg_probability"]
print ""
print "Characteristics of users in segment 2?"
print "-----------------------------------------------"
print "\n".join(['%s. %s' % (i + 1, x) for i, x in enumerate(segment["explanation"])])


Segment 2
---------------------------------------
Segment size      : 9.01 % of users
Churn probability : 0.560969559738

Characteristics of users in segment 2?
-----------------------------------------------
1. No "Quantity" events in the last 21 days
2. Sum of "Quantity" is greater than (or equal to) 109.50 each day in the last 90 days
3. Feature 'user_type' is 'extra-heavy'
4. Less than 2.50 days with an event in the last 90 days
5. Maximum events in a day the last 90 days less than 46.50
6. No "UnitPrice" events in the last 14 days

Understanding individual predictions: Why did the model make a prediction?


In [15]:
particular_user = valid[valid['CustomerID'] == '16200']
particular_user


Out[15]:
InvoiceDate InvoiceNo StockCode Description Quantity UnitPrice CustomerID
2011-06-28 15:47:00 558372 22847 BREAD BIN DINER STYLE
IVORY ...
2 16.95 16200
2011-06-28 15:47:00 558372 23287 RED VINTAGE SPOT BEAKER 4 0.85 16200
2011-06-28 15:47:00 558372 23288 GREEN VINTAGE SPOT BEAKER 4 0.85 16200
2011-06-28 15:47:00 558372 23285 PINK VINTAGE SPOT BEAKER 4 0.85 16200
2011-06-28 15:47:00 558372 23286 BLUE VINTAGE SPOT BEAKER 4 0.85 16200
2011-06-28 15:47:00 558372 23175 REGENCY MILK JUG PINK 2 3.25 16200
2011-06-28 15:47:00 558372 23322 LARGE WHITE HEART OF
WICKER ...
3 2.95 16200
2011-06-28 15:47:00 558372 22469 HEART OF WICKER SMALL 3 1.65 16200
2011-06-28 15:47:00 558372 23307 SET OF 60 PANTRY DESIGN
CAKE CASES ...
4 0.55 16200
2011-06-28 15:47:00 558372 22082 RIBBON REEL STRIPES
DESIGN ...
1 1.65 16200
[179 rows x 7 columns]
Note: Only the head of the TimeSeries is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.


In [16]:
explanations = model.explain(particular_user, user_data=users)


PROGRESS: Making a churn forecast for the time window:
PROGRESS: --------------------------------------------------
PROGRESS:  Start : 2011-12-05 14:40:00
PROGRESS:  End   : 2012-01-04 14:40:00
PROGRESS: --------------------------------------------------
PROGRESS: Grouping dataset by user.
PROGRESS: Resampling grouped observation_data by time-period 1 day, 0:00:00.
InvoiceNo is a categorical variable with too many different values (22061) and will be ignored.
StockCode is a categorical variable with too many different values (4058) and will be ignored.
PROGRESS: Generating features for boundary 2011-12-05 14:40:00.
Description is a categorical variable with too many different values (4195) and will be ignored.
PROGRESS: Joining user_data with aggregated features.

In [17]:
print ""
print "Model explanations"
print "---------------------------------------"
print 'Customer ID       : %s' % explanations["CustomerID"]
print 'Churn probability : %s' % explanations["probability"]
print ""
print "Why did the model make this prediction?"
print "---------------------------------------"
print "\n".join(['%s. %s' % (i + 1, x) for i, x in enumerate(explanations["explanation"][0])])


Model explanations
---------------------------------------
Customer ID       : ['16200']
Churn probability : [0.44427892565727234]

Why did the model make this prediction?
---------------------------------------
1. Greater than (or equal to) 89.50 events in the last 60 days
2. Less than 187.50 days since most recent event
3. Maximum events in a day the last 90 days greater than (or equal to) 46.50
4. Less than 2.50 days with an event in the last 90 days
5. Feature 'user_type' is 'extra-heavy'
6. Average number of events in the last 21 days greater than (or equal to) 106.50