In this notebook, we mainly utilize extreme gradient boost to improve the prediction model originially proposed in TLE 2016 November machine learning tuotrial. Extreme gradient boost can be viewed as an enhanced version of gradient boost by using a more regularized model formalization to control over-fitting, and XGB usually performs better. Applications of XGB can be found in many Kaggle competitions. Some recommended tutorrials can be found

Our work will be orginized in the follwing order:

•Background

•Exploratory Data Analysis

•Data Prepration and Model Selection

•Final Results

Background

The dataset we will use comes from a class excercise from The University of Kansas on Neural Networks and Fuzzy Systems. This exercise is based on a consortium project to use machine learning techniques to create a reservoir model of the largest gas fields in North America, the Hugoton and Panoma Fields. For more info on the origin of the data, see Bohling and Dubois (2003) and Dubois et al. (2007).

The dataset we will use is log data from nine wells that have been labeled with a facies type based on oberservation of core. We will use this log data to train a classifier to predict facies types.

This data is from the Council Grove gas reservoir in Southwest Kansas. The Panoma Council Grove Field is predominantly a carbonate gas reservoir encompassing 2700 square miles in Southwestern Kansas. This dataset is from nine wells (with 4149 examples), consisting of a set of seven predictor variables and a rock facies (class) for each example vector and validation (test) data (830 examples from two wells) having the same seven predictor variables in the feature vector. Facies are based on examination of cores from nine wells taken vertically at half-foot intervals. Predictor variables include five from wireline log measurements and two geologic constraining variables that are derived from geologic knowledge. These are essentially continuous variables sampled at a half-foot sample rate.

The seven predictor variables are: •Five wire line log curves include gamma ray (GR), resistivity logging (ILD_log10), photoelectric effect (PE), neutron-density porosity difference and average neutron-density porosity (DeltaPHI and PHIND). Note, some wells do not have PE. •Two geologic constraining variables: nonmarine-marine indicator (NM_M) and relative position (RELPOS)

The nine discrete facies (classes of rocks) are:

1.Nonmarine sandstone

2.Nonmarine coarse siltstone

3.Nonmarine fine siltstone

4.Marine siltstone and shale

5.Mudstone (limestone)

6.Wackestone (limestone)

7.Dolomite

8.Packstone-grainstone (limestone)

9.Phylloid-algal bafflestone (limestone)

These facies aren't discrete, and gradually blend into one another. Some have neighboring facies that are rather close. Mislabeling within these neighboring facies can be expected to occur. The following table lists the facies, their abbreviated labels and their approximate neighbors.

Facies/ Label/ Adjacent Facies

1 SS 2

2 CSiS 1,3

3 FSiS 2

4 SiSh 5

5 MS 4,6

6 WS 5,7

7 D 6,8

8 PS 6,7,9

9 BS 7,8

Exprolatory Data Analysis

After the background intorduction, we start to import the pandas library for some basic data analysis and manipulation. The matplotblib and seaborn are imported for data vislization.


In [1]:
%matplotlib inline
import pandas as pd
from pandas.tools.plotting import scatter_matrix
import matplotlib.pyplot as plt
import matplotlib as mpl
import seaborn as sns
import matplotlib.colors as colors

import xgboost as xgb
import numpy as np
from sklearn.metrics import confusion_matrix, f1_score, accuracy_score
from classification_utilities import display_cm, display_adj_cm
from sklearn.model_selection import GridSearchCV


from sklearn.model_selection import validation_curve
from sklearn.datasets import load_svmlight_files
from sklearn.model_selection import StratifiedKFold
from sklearn.datasets import make_classification
from xgboost.sklearn import XGBClassifier
from scipy.sparse import vstack

seed = 123
np.random.seed(seed)

In [2]:
import pandas as pd
filename = './facies_vectors.csv'
training_data = pd.read_csv(filename)
training_data.head(10)


Out[2]:
Facies Formation Well Name Depth GR ILD_log10 DeltaPHI PHIND PE NM_M RELPOS
0 3 A1 SH SHRIMPLIN 2793.0 77.45 0.664 9.9 11.915 4.6 1 1.000
1 3 A1 SH SHRIMPLIN 2793.5 78.26 0.661 14.2 12.565 4.1 1 0.979
2 3 A1 SH SHRIMPLIN 2794.0 79.05 0.658 14.8 13.050 3.6 1 0.957
3 3 A1 SH SHRIMPLIN 2794.5 86.10 0.655 13.9 13.115 3.5 1 0.936
4 3 A1 SH SHRIMPLIN 2795.0 74.58 0.647 13.5 13.300 3.4 1 0.915
5 3 A1 SH SHRIMPLIN 2795.5 73.97 0.636 14.0 13.385 3.6 1 0.894
6 3 A1 SH SHRIMPLIN 2796.0 73.72 0.630 15.6 13.930 3.7 1 0.872
7 3 A1 SH SHRIMPLIN 2796.5 75.65 0.625 16.5 13.920 3.5 1 0.830
8 3 A1 SH SHRIMPLIN 2797.0 73.79 0.624 16.2 13.980 3.4 1 0.809
9 3 A1 SH SHRIMPLIN 2797.5 76.89 0.615 16.9 14.220 3.5 1 0.787

In [3]:
training_data['Well Name'] = training_data['Well Name'].astype('category')
training_data['Formation'] = training_data['Formation'].astype('category')
training_data.info()


<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4149 entries, 0 to 4148
Data columns (total 11 columns):
Facies       4149 non-null int64
Formation    4149 non-null category
Well Name    4149 non-null category
Depth        4149 non-null float64
GR           4149 non-null float64
ILD_log10    4149 non-null float64
DeltaPHI     4149 non-null float64
PHIND        4149 non-null float64
PE           3232 non-null float64
NM_M         4149 non-null int64
RELPOS       4149 non-null float64
dtypes: category(2), float64(7), int64(2)
memory usage: 300.1 KB

In [4]:
training_data.describe()


C:\Users\zlc\Anaconda3\lib\site-packages\numpy\lib\function_base.py:3834: RuntimeWarning: Invalid value encountered in percentile
  RuntimeWarning)
Out[4]:
Facies Depth GR ILD_log10 DeltaPHI PHIND PE NM_M RELPOS
count 4149.000000 4149.000000 4149.000000 4149.000000 4149.000000 4149.000000 3232.000000 4149.000000 4149.000000
mean 4.503254 2906.867438 64.933985 0.659566 4.402484 13.201066 3.725014 1.518438 0.521852
std 2.474324 133.300164 30.302530 0.252703 5.274947 7.132846 0.896152 0.499720 0.286644
min 1.000000 2573.500000 10.149000 -0.025949 -21.832000 0.550000 0.200000 1.000000 0.000000
25% 2.000000 2821.500000 44.730000 0.498000 1.600000 8.500000 NaN 1.000000 0.277000
50% 4.000000 2932.500000 64.990000 0.639000 4.300000 12.020000 NaN 2.000000 0.528000
75% 6.000000 3007.000000 79.438000 0.822000 7.500000 16.050000 NaN 2.000000 0.769000
max 9.000000 3138.000000 361.150000 1.800000 19.312000 84.400000 8.094000 2.000000 1.000000

In [5]:
facies_colors = ['#F4D03F', '#F5B041','#DC7633','#6E2C00','#1B4F72',
                 '#2E86C1', '#AED6F1', '#A569BD', '#196F3D']

facies_labels = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS','WS', 'D','PS', 'BS']

facies_counts = training_data['Facies'].value_counts().sort_index()
facies_counts.index = facies_labels
facies_counts.plot(kind='bar',color=facies_colors,title='Distribution of Training Data by Facies')


Out[5]:
<matplotlib.axes._subplots.AxesSubplot at 0x19f9f740dd8>

In [6]:
sns.heatmap(training_data.corr(), vmax=1.0, square=True)


Out[6]:
<matplotlib.axes._subplots.AxesSubplot at 0x19f9f763b38>

Data Preparation and Model Selection

Now we are ready to test the XGB approach, and will use confusion matrix and f1_score, which were imported, as metric for classification, as well as GridSearchCV, which is an excellent tool for parameter optimization.


In [7]:
import xgboost as xgb
X_train = training_data.drop(['Facies', 'Well Name','Formation','Depth'], axis = 1 ) 
Y_train = training_data['Facies' ] - 1
dtrain = xgb.DMatrix(X_train, Y_train)

In [9]:
train = X_train.copy()

In [10]:
train['Facies']=Y_train

In [11]:
train.head()


Out[11]:
GR ILD_log10 DeltaPHI PHIND PE NM_M RELPOS Facies
0 77.45 0.664 9.9 11.915 4.6 1 1.000 2
1 78.26 0.661 14.2 12.565 4.1 1 0.979 2
2 79.05 0.658 14.8 13.050 3.6 1 0.957 2
3 86.10 0.655 13.9 13.115 3.5 1 0.936 2
4 74.58 0.647 13.5 13.300 3.4 1 0.915 2

The accuracy function and accuracy_adjacent function are defined in the following to quatify the prediction correctness.


In [12]:
def accuracy(conf):
    total_correct = 0.
    nb_classes = conf.shape[0]
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
    acc = total_correct/sum(sum(conf))
    return acc

adjacent_facies = np.array([[1], [0,2], [1], [4], [3,5], [4,6,7], [5,7], [5,6,8], [6,7]])

def accuracy_adjacent(conf, adjacent_facies):
    nb_classes = conf.shape[0]
    total_correct = 0.
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
        for j in adjacent_facies[i]:
            total_correct += conf[i][j]
    return total_correct / sum(sum(conf))

In [13]:
target='Facies'

Before processing further, we define a functin which will help us create XGBoost models and perform cross-validation.


In [14]:
def modelfit(alg, dtrain, features, useTrainCV=True,
             cv_fold=10,early_stopping_rounds = 50):
    if useTrainCV:
        xgb_param = alg.get_xgb_params()
        xgb_param['num_class']=9
        xgtrain = xgb.DMatrix(train[features].values,label = train[target].values)
        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=
                          alg.get_params()['n_estimators'],nfold=cv_fold,
                          metrics='merror',early_stopping_rounds = early_stopping_rounds)
        alg.set_params(n_estimators=cvresult.shape[0])
        
    #Fit the algorithm on the data
    alg.fit(dtrain[features], dtrain[target],eval_metric='merror')
        
    #Predict training set:
    dtrain_prediction = alg.predict(dtrain[features])
    dtrain_predprob = alg.predict_proba(dtrain[features])[:,1]
        
    #Pring model report
    print ("\nModel Report")
    print ("Accuracy : %.4g" % accuracy_score(dtrain[target], 
                                            dtrain_prediction))
    print ("F1 score (Train) : %f" % f1_score(dtrain[target], 
                                            dtrain_prediction,average='weighted'))
    feat_imp = pd.Series(alg.booster().get_fscore()).sort_values(ascending=False)
    feat_imp.plot(kind='bar',title='Feature Importances')
    plt.ylabel('Feature Importance Score')

In [15]:
features =[x for x in X_train.columns]
features


Out[15]:
['GR', 'ILD_log10', 'DeltaPHI', 'PHIND', 'PE', 'NM_M', 'RELPOS']

General Approach for Parameter Tuning

We are going to preform the steps as follows:

1.Choose a relatively high learning rate, e.g., 0.1. Usually somewhere between 0.05 and 0.3 should work for different problems.

2.Determine the optimum number of tress for this learning rate.XGBoost has a very usefull function called as "cv" which performs cross-validation at each boosting iteration and thus returns the optimum number of tress required.

3.Tune tree-based parameters(max_depth, min_child_weight, gamma, subsample, colsample_bytree) for decided learning rate and number of trees.

4.Tune regularization parameters(lambda, alpha) for xgboost which can help reduce model complexity and enhance performance.

5.Lower the learning rate and decide the optimal parameters.

Step 1:Fix learning rate and number of estimators for tuning tree-based parameters

In order to decide on boosting parameters, we need to set some initial values of other parameters. Lets take the following values:

1.max_depth = 5

2.min_child_weight = 1

3.gamma = 0

4.subsample, colsample_bytree = 0.8 : This is a commonly used used start value.

5.scale_pos_weight = 1

Please note that all the above are just initial estimates and will be tuned later. Lets take the default learning rate of 0.1 here and check the optimum number of trees using cv function of xgboost. The function defined above will do it for us.


In [15]:
from xgboost import XGBClassifier
xgb1 = XGBClassifier(
    learning_rate = 0.1,
    n_estimators=1000,
    max_depth=5,
    min_child_weight=1,
    gamma = 0,
    subsample=0.8,
    colsample_bytree=0.8,
    objective='multi:softmax',
    nthread =4,
    seed = 123,
)

In [16]:
modelfit(xgb1, train, features)


Model Report
Accuracy : 0.9978
F1 score (Train) : 0.997831

In [17]:
xgb1


Out[17]:
XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=5,
       min_child_weight=1, missing=None, n_estimators=370, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123, silent=True, subsample=0.8)

Step 2: Tune max_depth and min_child_weight


In [18]:
from sklearn.model_selection import GridSearchCV
param_test1={
    'max_depth':range(3,10,2),
    'min_child_weight':range(1,6,2)
}

gs1 = GridSearchCV(xgb1,param_grid=param_test1, 
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs1.fit(train[features],train[target])
gs1.grid_scores_, gs1.best_params_,gs1.best_score_


C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\model_selection\_search.py:667: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20
  DeprecationWarning)
Out[18]:
([mean: 0.51984, std: 0.06464, params: {'max_depth': 3, 'min_child_weight': 1},
  mean: 0.51766, std: 0.06520, params: {'max_depth': 3, 'min_child_weight': 3},
  mean: 0.52031, std: 0.06484, params: {'max_depth': 3, 'min_child_weight': 5},
  mean: 0.52175, std: 0.06474, params: {'max_depth': 5, 'min_child_weight': 1},
  mean: 0.51813, std: 0.07043, params: {'max_depth': 5, 'min_child_weight': 3},
  mean: 0.51982, std: 0.06751, params: {'max_depth': 5, 'min_child_weight': 5},
  mean: 0.51741, std: 0.07279, params: {'max_depth': 7, 'min_child_weight': 1},
  mean: 0.51667, std: 0.06840, params: {'max_depth': 7, 'min_child_weight': 3},
  mean: 0.51572, std: 0.06637, params: {'max_depth': 7, 'min_child_weight': 5},
  mean: 0.52077, std: 0.06725, params: {'max_depth': 9, 'min_child_weight': 1},
  mean: 0.51716, std: 0.06321, params: {'max_depth': 9, 'min_child_weight': 3},
  mean: 0.51861, std: 0.06685, params: {'max_depth': 9, 'min_child_weight': 5}],
 {'max_depth': 5, 'min_child_weight': 1},
 0.52175269097096377)

In [17]:
param_test2={
    'max_depth':[8,9,10],
    'min_child_weight':[1,2]
}

gs2 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=5,
       min_child_weight=1, n_estimators=290, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.8),param_grid=param_test2, 
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs2.fit(train[features],train[target])
gs2.grid_scores_, gs2.best_params_,gs2.best_score_


/Users/littleni/anaconda/lib/python3.5/site-packages/sklearn/model_selection/_search.py:667: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20
  DeprecationWarning)
Out[17]:
([mean: 0.52391, std: 0.06883, params: {'min_child_weight': 1, 'max_depth': 8},
  mean: 0.52078, std: 0.07002, params: {'min_child_weight': 2, 'max_depth': 8},
  mean: 0.52584, std: 0.06572, params: {'min_child_weight': 1, 'max_depth': 9},
  mean: 0.51621, std: 0.06604, params: {'min_child_weight': 2, 'max_depth': 9},
  mean: 0.52222, std: 0.06617, params: {'min_child_weight': 1, 'max_depth': 10},
  mean: 0.52222, std: 0.06944, params: {'min_child_weight': 2, 'max_depth': 10}],
 {'max_depth': 9, 'min_child_weight': 1},
 0.52584156512733504)

In [18]:
gs2.best_estimator_


Out[18]:
XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, missing=None, n_estimators=290, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123, silent=True, subsample=0.8)

Step 3: Tune gamma


In [19]:
param_test3={
   'gamma':[i/10.0 for i in range(0,5)]
}

gs3 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=370, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.8),param_grid=param_test3, 
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs3.fit(train[features],train[target])
gs3.grid_scores_, gs3.best_params_,gs3.best_score_


/Users/littleni/anaconda/lib/python3.5/site-packages/sklearn/model_selection/_search.py:667: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20
  DeprecationWarning)
Out[19]:
([mean: 0.52584, std: 0.06572, params: {'gamma': 0.0},
  mean: 0.52632, std: 0.07086, params: {'gamma': 0.1},
  mean: 0.52656, std: 0.07059, params: {'gamma': 0.2},
  mean: 0.52633, std: 0.06713, params: {'gamma': 0.3},
  mean: 0.52440, std: 0.06480, params: {'gamma': 0.4}],
 {'gamma': 0.2},
 0.52655662535367609)

In [20]:
xgb2 = XGBClassifier(
    learning_rate = 0.1,
    n_estimators=1000,
    max_depth=9,
    min_child_weight=1,
    gamma = 0.2,
    subsample=0.8,
    colsample_bytree=0.8,
    objective='multi:softmax',
    nthread =4,
    scale_pos_weight=1,
    seed = seed,
)
modelfit(xgb2,train,features)


Model Report
Accuracy : 0.9978
F1 score (Train) : 0.997830

In [21]:
xgb2


Out[21]:
XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, missing=None, n_estimators=236, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123, silent=True, subsample=0.8)

Step 4:Tune subsample and colsample_bytree


In [22]:
param_test4={
    'subsample':[i/10.0 for i in range(6,10)],
    'colsample_bytree':[i/10.0 for i in range(6,10)]
}

gs4 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=236, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.8),param_grid=param_test4, 
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs4.fit(train[features],train[target])
gs4.grid_scores_, gs4.best_params_,gs4.best_score_


/Users/littleni/anaconda/lib/python3.5/site-packages/sklearn/model_selection/_search.py:667: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20
  DeprecationWarning)
Out[22]:
([mean: 0.52222, std: 0.07060, params: {'colsample_bytree': 0.6, 'subsample': 0.6},
  mean: 0.52392, std: 0.06317, params: {'colsample_bytree': 0.6, 'subsample': 0.7},
  mean: 0.52994, std: 0.06877, params: {'colsample_bytree': 0.6, 'subsample': 0.8},
  mean: 0.52704, std: 0.06807, params: {'colsample_bytree': 0.6, 'subsample': 0.9},
  mean: 0.52222, std: 0.07060, params: {'colsample_bytree': 0.7, 'subsample': 0.6},
  mean: 0.52392, std: 0.06317, params: {'colsample_bytree': 0.7, 'subsample': 0.7},
  mean: 0.52994, std: 0.06877, params: {'colsample_bytree': 0.7, 'subsample': 0.8},
  mean: 0.52704, std: 0.06807, params: {'colsample_bytree': 0.7, 'subsample': 0.9},
  mean: 0.53162, std: 0.07243, params: {'colsample_bytree': 0.8, 'subsample': 0.6},
  mean: 0.52705, std: 0.07100, params: {'colsample_bytree': 0.8, 'subsample': 0.7},
  mean: 0.52608, std: 0.07036, params: {'colsample_bytree': 0.8, 'subsample': 0.8},
  mean: 0.51885, std: 0.06818, params: {'colsample_bytree': 0.8, 'subsample': 0.9},
  mean: 0.52126, std: 0.07021, params: {'colsample_bytree': 0.9, 'subsample': 0.6},
  mean: 0.52077, std: 0.06890, params: {'colsample_bytree': 0.9, 'subsample': 0.7},
  mean: 0.51908, std: 0.06919, params: {'colsample_bytree': 0.9, 'subsample': 0.8},
  mean: 0.52006, std: 0.07087, params: {'colsample_bytree': 0.9, 'subsample': 0.9}],
 {'colsample_bytree': 0.8, 'subsample': 0.6},
 0.53161835175244798)

In [23]:
param_test4b={
    'subsample':[i/10.0 for i in range(5,7)],
}

gs4b = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=236, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.8),param_grid=param_test4b, 
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs4b.fit(train[features],train[target])
gs4b.grid_scores_, gs4b.best_params_,gs4b.best_score_


/Users/littleni/anaconda/lib/python3.5/site-packages/sklearn/model_selection/_search.py:667: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20
  DeprecationWarning)
Out[23]:
([mean: 0.52414, std: 0.06889, params: {'subsample': 0.5},
  mean: 0.53162, std: 0.07243, params: {'subsample': 0.6}],
 {'subsample': 0.6},
 0.53161835175244798)

Step 5: Tuning Regularization Parameters


In [24]:
param_test5={
    'reg_alpha':[1e-5, 1e-2, 0.1, 1, 100]
}

gs5 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=236, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.6),param_grid=param_test5, 
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs5.fit(train[features],train[target])
gs5.grid_scores_, gs5.best_params_,gs5.best_score_


/Users/littleni/anaconda/lib/python3.5/site-packages/sklearn/model_selection/_search.py:667: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20
  DeprecationWarning)
Out[24]:
([mean: 0.52825, std: 0.07059, params: {'reg_alpha': 1e-05},
  mean: 0.52513, std: 0.07066, params: {'reg_alpha': 0.01},
  mean: 0.52800, std: 0.07177, params: {'reg_alpha': 0.1},
  mean: 0.52462, std: 0.07812, params: {'reg_alpha': 1},
  mean: 0.47094, std: 0.03632, params: {'reg_alpha': 100}],
 {'reg_alpha': 1e-05},
 0.52825352135380199)

In [25]:
param_test6={
    'reg_alpha':[0, 0.001, 0.005, 0.01, 0.05]
}

gs6 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=236, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.6),param_grid=param_test6, 
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs6.fit(train[features],train[target])
gs6.grid_scores_, gs6.best_params_,gs6.best_score_


/Users/littleni/anaconda/lib/python3.5/site-packages/sklearn/model_selection/_search.py:667: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20
  DeprecationWarning)
Out[25]:
([mean: 0.53162, std: 0.07243, params: {'reg_alpha': 0},
  mean: 0.52968, std: 0.07384, params: {'reg_alpha': 0.001},
  mean: 0.52921, std: 0.07140, params: {'reg_alpha': 0.005},
  mean: 0.52513, std: 0.07066, params: {'reg_alpha': 0.01},
  mean: 0.53186, std: 0.07231, params: {'reg_alpha': 0.05}],
 {'reg_alpha': 0.05},
 0.53185612242083324)

In [26]:
xgb3 = XGBClassifier(
    learning_rate = 0.1,
    n_estimators=1000,
    max_depth=9,
    min_child_weight=1,
    gamma = 0.2,
    subsample=0.6,
    colsample_bytree=0.8,
    reg_alpha=0.05,
    objective='multi:softmax',
    nthread =4,
    scale_pos_weight=1,
    seed = seed,
)
modelfit(xgb3,train,features)


Model Report
Accuracy : 0.9966
F1 score (Train) : 0.996622

In [27]:
xgb3


Out[27]:
XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, missing=None, n_estimators=122, nthread=4,
       objective='multi:softprob', reg_alpha=0.05, reg_lambda=1,
       scale_pos_weight=1, seed=123, silent=True, subsample=0.6)

In [8]:
model = XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, missing=None, n_estimators=122, nthread=4,
       objective='multi:softprob', reg_alpha=0.05, reg_lambda=1,
       scale_pos_weight=1, seed=123, silent=True, subsample=0.6)
model.fit(X_train, Y_train)
xgb.plot_importance(model)


Out[8]:
<matplotlib.axes._subplots.AxesSubplot at 0x26c15d3a8d0>

In [ ]:

Step 6: Reducing Learning Rate


In [28]:
xgb4 = XGBClassifier(
    learning_rate = 0.01,
    n_estimators=5000,
    max_depth=9,
    min_child_weight=1,
    gamma = 0.2,
    subsample=0.6,
    colsample_bytree=0.8,
    reg_alpha=0.05,
    objective='multi:softmax',
    nthread =4,
    scale_pos_weight=1,
    seed = seed,
)
modelfit(xgb4,train,features)


Model Report
Accuracy : 0.9467
F1 score (Train) : 0.946662

In [29]:
xgb4


Out[29]:
XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.01, max_delta_step=0, max_depth=9,
       min_child_weight=1, missing=None, n_estimators=432, nthread=4,
       objective='multi:softprob', reg_alpha=0.05, reg_lambda=1,
       scale_pos_weight=1, seed=123, silent=True, subsample=0.6)

Cross Validation

Next we use our tuned final model to do cross validation on the training data set. One of the wells will be used as test data and the rest will be the training data. Each iteration, a different well is chosen.


In [5]:
# Load data 
filename = './facies_vectors.csv'
data = pd.read_csv(filename)

# Change to category data type
data['Well Name'] = data['Well Name'].astype('category')
data['Formation'] = data['Formation'].astype('category')

# Leave one well out for cross validation 
well_names = data['Well Name'].unique()
f1=[]
for i in range(len(well_names)):
    
    # Split data for training and testing
    X_train = data.drop(['Facies', 'Formation','Depth'], axis = 1 ) 
    Y_train = data['Facies' ] - 1
    
    train_X = X_train[X_train['Well Name'] != well_names[i] ]
    train_Y = Y_train[X_train['Well Name'] != well_names[i] ]
    test_X  = X_train[X_train['Well Name'] == well_names[i] ]
    test_Y  = Y_train[X_train['Well Name'] == well_names[i] ]

    train_X = train_X.drop(['Well Name'], axis = 1 ) 
    test_X = test_X.drop(['Well Name'], axis = 1 )

    # Final recommended model based on the extensive parameters search
    model_final = XGBClassifier(base_score=0.5, colsample_bylevel=1,
       colsample_bytree=0.8, gamma=0.2,
       learning_rate=0.01, max_delta_step=0, max_depth=9,
       min_child_weight=1, missing=None, n_estimators=432, nthread=4,
       objective='multi:softmax', reg_alpha=0.05, reg_lambda=1,
       scale_pos_weight=1, seed=123, silent=1,
       subsample=0.6)
    # Train the model based on training data
    model_final.fit( train_X , train_Y , eval_metric = 'merror' )


    # Predict on the test set
    predictions = model_final.predict(test_X)

    # Print report
    print ("\n------------------------------------------------------")
    print ("Validation on the leaving out well " + well_names[i])
    conf = confusion_matrix( test_Y, predictions, labels = np.arange(9) )
    print ("\nModel Report")
    print ("-Accuracy: %.6f" % ( accuracy(conf) ))
    print ("-Adjacent Accuracy: %.6f" % ( accuracy_adjacent(conf, adjacent_facies) ))
    print ("-F1 Score: %.6f" % ( f1_score ( test_Y , predictions , labels = np.arange(9), average = 'weighted' ) ))
    f1.append(f1_score ( test_Y , predictions , labels = np.arange(9), average = 'weighted' ))
    facies_labels = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS',
                     'WS', 'D','PS', 'BS']
    print ("\nConfusion Matrix Results")
    from classification_utilities import display_cm, display_adj_cm
    display_cm(conf, facies_labels,display_metrics=True, hide_zeros=True)
    
print ("\n------------------------------------------------------")
print ("Final Results")
print ("-Average F1 Score: %6f" % (sum(f1)/(1.0*len(f1))))


------------------------------------------------------
Validation on the leaving out well SHRIMPLIN

Model Report
-Accuracy: 0.607219
-Adjacent Accuracy: 0.951168
-F1 Score: 0.583496

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS                                                           0
     CSiS     1    97    20                                       118
     FSiS     1    53    69                                       123
     SiSh                      12           2           4          18
       MS                       6     8    39          10          63
       WS                       3     1    36     2    19     2    63
        D                                   1           4           5
       PS                             1    10          52     6    69
       BS                                                    12    12

Precision  0.00  0.65  0.78  0.57  0.80  0.41  0.00  0.58  0.60  0.65
   Recall  0.00  0.82  0.56  0.67  0.13  0.57  0.00  0.75  1.00  0.61
       F1  0.00  0.72  0.65  0.62  0.22  0.48  0.00  0.66  0.75  0.58
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1115: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true samples.
  'recall', 'true', average, warn_for)
------------------------------------------------------
Validation on the leaving out well ALEXANDER D

Model Report
-Accuracy: 0.628755
-Adjacent Accuracy: 0.912017
-F1 Score: 0.594002

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS                                                           0
     CSiS          82    35                                       117
     FSiS          16    75                                        91
     SiSh                      39                 3     2          44
       MS                       3    13                10          26
       WS                      13    19     2     7    28          69
        D                             2           8     6          16
       PS                       7     4     4    11    69     3    98
       BS                                                     5     5

Precision  0.00  0.84  0.68  0.63  0.34  0.33  0.28  0.60  0.62  0.61
   Recall  0.00  0.70  0.82  0.89  0.50  0.03  0.50  0.70  1.00  0.63
       F1  0.00  0.76  0.75  0.74  0.41  0.05  0.36  0.65  0.77  0.59
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.
  'precision', 'predicted', average, warn_for)
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1115: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true samples.
  'recall', 'true', average, warn_for)
------------------------------------------------------
Validation on the leaving out well SHANKLE

Model Report
-Accuracy: 0.505568
-Adjacent Accuracy: 0.966592
-F1 Score: 0.481789

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS     6    83                                              89
     CSiS     7    72    10                                        89
     FSiS          48    68                             1         117
     SiSh                       3           3           1           7
       MS                      14     1     2     1     1          19
       WS                 2     6     4    46          13          71
        D                                   2     9     6          17
       PS                                  17     1    22          40
       BS                                                           0

Precision  0.46  0.35  0.85  0.13  0.20  0.66  0.82  0.50  0.00  0.57
   Recall  0.07  0.81  0.58  0.43  0.05  0.65  0.53  0.55  0.00  0.51
       F1  0.12  0.49  0.69  0.20  0.08  0.65  0.64  0.52  0.00  0.48
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.
  'precision', 'predicted', average, warn_for)
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1115: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true samples.
  'recall', 'true', average, warn_for)
------------------------------------------------------
Validation on the leaving out well LUKE G U

Model Report
-Accuracy: 0.629067
-Adjacent Accuracy: 0.934924
-F1 Score: 0.658732

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS                                                           0
     CSiS    14    86    17                                       117
     FSiS     5    42    78                             4         129
     SiSh                      29     3     3                      35
       MS                             1                 1           2
       WS                       7    23    39          13     2    84
        D                                   1     8    11          20
       PS                 2     2     4    16     1    49          74
       BS                                                           0

Precision  0.00  0.67  0.80  0.76  0.03  0.66  0.89  0.63  0.00  0.71
   Recall  0.00  0.74  0.60  0.83  0.50  0.46  0.40  0.66  0.00  0.63
       F1  0.00  0.70  0.69  0.79  0.06  0.55  0.55  0.64  0.00  0.66
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1115: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true samples.
  'recall', 'true', average, warn_for)
------------------------------------------------------
Validation on the leaving out well KIMZEY A

Model Report
-Accuracy: 0.523918
-Adjacent Accuracy: 0.892938
-F1 Score: 0.490607

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS           5     4                                         9
     CSiS          73    12                                        85
     FSiS          38    36                                        74
     SiSh                      26     2    15                      43
       MS                       5     1    31          14     2    53
       WS                 1     4          26     1    17     2    51
        D                       2           6     5    13     1    27
       PS                       1          25     4    58     2    90
       BS                                   1           1     5     7

Precision  0.00  0.63  0.68  0.68  0.33  0.25  0.50  0.56  0.42  0.53
   Recall  0.00  0.86  0.49  0.60  0.02  0.51  0.19  0.64  0.71  0.52
       F1  0.00  0.73  0.57  0.64  0.04  0.34  0.27  0.60  0.53  0.49
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.
  'precision', 'predicted', average, warn_for)
------------------------------------------------------
Validation on the leaving out well CROSS H CATTLE

Model Report
-Accuracy: 0.391218
-Adjacent Accuracy: 0.882236
-F1 Score: 0.380480

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS    35   106    17                                       158
     CSiS     2    63    76                             1         142
     FSiS           5    39                 1           2          47
     SiSh                 4     7          13           1          25
       MS           4     3           1    16     1     3          28
       WS                       1          22     1     7          31
        D                                         1     1           2
       PS                 4     3     1    27     5    28          68
       BS                                                           0

Precision  0.95  0.35  0.27  0.64  0.50  0.28  0.12  0.65  0.00  0.59
   Recall  0.22  0.44  0.83  0.28  0.04  0.71  0.50  0.41  0.00  0.39
       F1  0.36  0.39  0.41  0.39  0.07  0.40  0.20  0.50  0.00  0.38
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.
  'precision', 'predicted', average, warn_for)
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1115: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true samples.
  'recall', 'true', average, warn_for)
------------------------------------------------------
Validation on the leaving out well NOLAN

Model Report
-Accuracy: 0.498795
-Adjacent Accuracy: 0.860241
-F1 Score: 0.524120

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS           4                                               4
     CSiS    17    82    18                             1         118
     FSiS     3    23    39                 1           2          68
     SiSh                 1     6     2    17           2          28
       MS           3                 3    27     2     9     3    47
       WS     1                 1          11          14     3    30
        D                       1                 2     1           4
       PS                 5     3          18     3    64    23   116
       BS                                                           0

Precision  0.00  0.73  0.62  0.55  0.60  0.15  0.29  0.69  0.00  0.62
   Recall  0.00  0.69  0.57  0.21  0.06  0.37  0.50  0.55  0.00  0.50
       F1  0.00  0.71  0.60  0.31  0.12  0.21  0.36  0.61  0.00  0.52
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1115: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true samples.
  'recall', 'true', average, warn_for)
------------------------------------------------------
Validation on the leaving out well Recruit F9

Model Report
-Accuracy: 0.787500
-Adjacent Accuracy: 0.950000
-F1 Score: 0.881119

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS                                                           0
     CSiS                                                           0
     FSiS                                                           0
     SiSh                                                           0
       MS                                                           0
       WS                                                           0
        D                                                           0
       PS                                                           0
       BS                                   4     2    11    63    80

Precision  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  1.00  1.00
   Recall  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.79  0.79
       F1  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.88  0.88
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.
  'precision', 'predicted', average, warn_for)
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1115: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true samples.
  'recall', 'true', average, warn_for)
------------------------------------------------------
Validation on the leaving out well NEWBY

Model Report
-Accuracy: 0.481641
-Adjacent Accuracy: 0.892009
-F1 Score: 0.478244

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS                                                           0
     CSiS    16    59    21                             2          98
     FSiS     1    33    45                             1          80
     SiSh                      33     4    13     4     4          58
       MS                 4     2          11     4     7          28
       WS                       3    18    33     3    39          96
        D                                   2     6     8          16
       PS                             2     8          46          56
       BS                       1           4          25     1    31

Precision  0.00  0.64  0.64  0.85  0.00  0.46  0.35  0.35  1.00  0.57
   Recall  0.00  0.60  0.56  0.57  0.00  0.34  0.38  0.82  0.03  0.48
       F1  0.00  0.62  0.60  0.68  0.00  0.40  0.36  0.49  0.06  0.48
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1115: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true samples.
  'recall', 'true', average, warn_for)
------------------------------------------------------
Validation on the leaving out well CHURCHMAN BIBLE

Model Report
-Accuracy: 0.594059
-Adjacent Accuracy: 0.888614
-F1 Score: 0.567658

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS           7     1                                         8
     CSiS     3    32    20                 1                      56
     FSiS          10    37     2           1           1          51
     SiSh                       7     1     5                      13
       MS                 1     2     3    20           4          30
       WS                      12          63          12          87
        D                 2     1     1     2     5    20     3    34
       PS                 3     3     4    21          43     1    75
       BS                                                    50    50

Precision  0.00  0.65  0.58  0.26  0.33  0.56  1.00  0.54  0.93  0.62
   Recall  0.00  0.57  0.73  0.54  0.10  0.72  0.15  0.57  1.00  0.59
       F1  0.00  0.61  0.64  0.35  0.15  0.63  0.26  0.55  0.96  0.57

------------------------------------------------------
Final Results
-Average F1 Score: 0.564025

Model from all data set


In [16]:
# Load data 
filename = './facies_vectors.csv'
data = pd.read_csv(filename)

# Change to category data type
data['Well Name'] = data['Well Name'].astype('category')
data['Formation'] = data['Formation'].astype('category')

# Split data for training and testing
X_train_all = data.drop(['Facies', 'Formation','Depth'], axis = 1 ) 
Y_train_all = data['Facies' ] - 1

X_train_all = X_train_all.drop(['Well Name'], axis = 1)

# Final recommended model based on the extensive parameters search
model_final = XGBClassifier(base_score=0.5, colsample_bylevel=1,
       colsample_bytree=0.8, gamma=0.2,
       learning_rate=0.01, max_delta_step=0, max_depth=9,
       min_child_weight=1, missing=None, n_estimators=432, nthread=4,
       objective='multi:softmax', reg_alpha=0.05, reg_lambda=1,
       scale_pos_weight=1, seed=123, silent=1,
       subsample=0.6)

# Train the model based on training data
model_final.fit(X_train_all , Y_train_all , eval_metric = 'merror' )


Out[16]:
XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.01, max_delta_step=0, max_depth=9,
       min_child_weight=1, missing=None, n_estimators=432, nthread=4,
       objective='multi:softprob', reg_alpha=0.05, reg_lambda=1,
       scale_pos_weight=1, seed=123, silent=1, subsample=0.6)

In [17]:
# Leave one well out for cross validation 
well_names = data['Well Name'].unique()
f1=[]

for i in range(len(well_names)):
    
    X_train = data.drop(['Facies', 'Formation','Depth'], axis = 1 ) 
    Y_train = data['Facies' ] - 1

    train_X = X_train[X_train['Well Name'] != well_names[i] ]
    train_Y = Y_train[X_train['Well Name'] != well_names[i] ]
    test_X  = X_train[X_train['Well Name'] == well_names[i] ]
    test_Y  = Y_train[X_train['Well Name'] == well_names[i] ]

    train_X = train_X.drop(['Well Name'], axis = 1 ) 
    test_X = test_X.drop(['Well Name'], axis = 1 )
    #print(test_Y)
    predictions = model_final.predict(test_X)
    
    # Print report
    print ("\n------------------------------------------------------")
    print ("Validation on the leaving out well " + well_names[i])
    conf = confusion_matrix( test_Y, predictions, labels = np.arange(9) )
    print ("\nModel Report")
    print ("-Accuracy: %.6f" % ( accuracy(conf) ))
    print ("-Adjacent Accuracy: %.6f" % ( accuracy_adjacent(conf, adjacent_facies) ))
    print ("-F1 Score: %.6f" % ( f1_score ( test_Y , predictions , labels = np.arange(9), average = 'weighted' ) ))
    f1.append(f1_score ( test_Y , predictions , labels = np.arange(9), average = 'weighted' ))
    facies_labels = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS',
                     'WS', 'D','PS', 'BS']
    print ("\nConfusion Matrix Results")
    from classification_utilities import display_cm, display_adj_cm
    display_cm(conf, facies_labels,display_metrics=True, hide_zeros=True)
    
print ("\n------------------------------------------------------")
print ("Final Results")
print ("-Average F1 Score: %6f" % (sum(f1)/(1.0*len(f1))))


------------------------------------------------------
Validation on the leaving out well SHRIMPLIN

Model Report
-Accuracy: 0.974522
-Adjacent Accuracy: 1.000000
-F1 Score: 0.974416

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS                                                           0
     CSiS         114     4                                       118
     FSiS           5   118                                       123
     SiSh                      18                                  18
       MS                            63                            63
       WS                                  63                      63
        D                                         4     1           5
       PS                                   2          67          69
       BS                                                    12    12

Precision  0.00  0.96  0.97  1.00  1.00  0.97  1.00  0.99  1.00  0.97
   Recall  0.00  0.97  0.96  1.00  1.00  1.00  0.80  0.97  1.00  0.97
       F1  0.00  0.96  0.96  1.00  1.00  0.98  0.89  0.98  1.00  0.97

------------------------------------------------------
Validation on the leaving out well ALEXANDER D

Model Report
-Accuracy: 0.952790
-Adjacent Accuracy: 0.980687
-F1 Score: 0.952061

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS                                                           0
     CSiS         114     3                                       117
     FSiS           3    88                                        91
     SiSh                      43           1                      44
       MS                       1    19     3           3          26
       WS                       1          65           3          69
        D                             1          15                16
       PS                       3                      95          98
       BS                                                     5     5

Precision  0.00  0.97  0.97  0.90  0.95  0.94  1.00  0.94  1.00  0.95
   Recall  0.00  0.97  0.97  0.98  0.73  0.94  0.94  0.97  1.00  0.95
       F1  0.00  0.97  0.97  0.93  0.83  0.94  0.97  0.95  1.00  0.95

------------------------------------------------------
Validation on the leaving out well SHANKLE

Model Report
-Accuracy: 0.968820
-Adjacent Accuracy: 0.993318
-F1 Score: 0.968757

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS    85     4                                              89
     CSiS          88     1                                        89
     FSiS           2   115                                       117
     SiSh                       5                       2           7
       MS                            19                            19
       WS                 1                67           3          71
        D                                        17                17
       PS                                   1          39          40
       BS                                                           0

Precision  1.00  0.94  0.98  1.00  1.00  0.99  1.00  0.89  0.00  0.97
   Recall  0.96  0.99  0.98  0.71  1.00  0.94  1.00  0.97  0.00  0.97
       F1  0.98  0.96  0.98  0.83  1.00  0.96  1.00  0.93  0.00  0.97
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.
  'precision', 'predicted', average, warn_for)
C:\Users\zlc\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1115: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true samples.
  'recall', 'true', average, warn_for)
------------------------------------------------------
Validation on the leaving out well LUKE G U

Model Report
-Accuracy: 0.954447
-Adjacent Accuracy: 0.995662
-F1 Score: 0.954290

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS                                                           0
     CSiS         114     3                                       117
     FSiS           6   123                                       129
     SiSh                      34           1                      35
       MS                             2                             2
       WS                                  83           1          84
        D                                        20                20
       PS                       1           9          64          74
       BS                                                           0

Precision  0.00  0.95  0.98  0.97  1.00  0.89  1.00  0.98  0.00  0.96
   Recall  0.00  0.97  0.95  0.97  1.00  0.99  1.00  0.86  0.00  0.95
       F1  0.00  0.96  0.96  0.97  1.00  0.94  1.00  0.92  0.00  0.95

------------------------------------------------------
Validation on the leaving out well KIMZEY A

Model Report
-Accuracy: 0.940774
-Adjacent Accuracy: 0.977221
-F1 Score: 0.938915

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS     4     1     4                                         9
     CSiS          83     2                                        85
     FSiS           8    66                                        74
     SiSh                      43                                  43
       MS                       1    51     1                      53
       WS                 1     3          46           1          51
        D                       1                25     1          27
       PS                             1     1          88          90
       BS                                                     7     7

Precision  1.00  0.90  0.90  0.90  0.98  0.96  1.00  0.98  1.00  0.94
   Recall  0.44  0.98  0.89  1.00  0.96  0.90  0.93  0.98  1.00  0.94
       F1  0.62  0.94  0.90  0.95  0.97  0.93  0.96  0.98  1.00  0.94

------------------------------------------------------
Validation on the leaving out well CROSS H CATTLE

Model Report
-Accuracy: 0.926148
-Adjacent Accuracy: 0.980040
-F1 Score: 0.927114

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS   139    17     2                                       158
     CSiS     1   137     4                                       142
     FSiS           2    45                                        47
     SiSh           1     1    23                                  25
       MS           1     2          23     2                      28
       WS                                  31                      31
        D                                         2                 2
       PS                 3                 1          64          68
       BS                                                           0

Precision  0.99  0.87  0.79  1.00  1.00  0.91  1.00  1.00  0.00  0.93
   Recall  0.88  0.96  0.96  0.92  0.82  1.00  1.00  0.94  0.00  0.93
       F1  0.93  0.91  0.87  0.96  0.90  0.95  1.00  0.97  0.00  0.93

------------------------------------------------------
Validation on the leaving out well NOLAN

Model Report
-Accuracy: 0.925301
-Adjacent Accuracy: 0.968675
-F1 Score: 0.923646

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS     1     3                                               4
     CSiS         116     1                 1                     118
     FSiS           5    61                 1           1          68
     SiSh                      25           2           1          28
       MS           2     1          39     4           1          47
       WS     1                            25           4          30
        D                                         3     1           4
       PS                 2                           114         116
       BS                                                           0

Precision  0.50  0.92  0.94  1.00  1.00  0.76  1.00  0.93  0.00  0.93
   Recall  0.25  0.98  0.90  0.89  0.83  0.83  0.75  0.98  0.00  0.93
       F1  0.33  0.95  0.92  0.94  0.91  0.79  0.86  0.96  0.00  0.92

------------------------------------------------------
Validation on the leaving out well Recruit F9

Model Report
-Accuracy: 0.975000
-Adjacent Accuracy: 0.987500
-F1 Score: 0.987342

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS                                                           0
     CSiS                                                           0
     FSiS                                                           0
     SiSh                                                           0
       MS                                                           0
       WS                                                           0
        D                                                           0
       PS                                                           0
       BS                                   1           1    78    80

Precision  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  1.00  1.00
   Recall  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.97  0.97
       F1  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.99  0.99

------------------------------------------------------
Validation on the leaving out well NEWBY

Model Report
-Accuracy: 0.939525
-Adjacent Accuracy: 0.989201
-F1 Score: 0.939183

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS                                                           0
     CSiS          94     4                                        98
     FSiS           7    73                                        80
     SiSh                      56           1           1          58
       MS                       1    22     5                      28
       WS                             1    92           3          96
        D                                        15     1          16
       PS                       1           1          54          56
       BS                                   2                29    31

Precision  0.00  0.93  0.95  0.97  0.96  0.91  1.00  0.92  1.00  0.94
   Recall  0.00  0.96  0.91  0.97  0.79  0.96  0.94  0.96  0.94  0.94
       F1  0.00  0.94  0.93  0.97  0.86  0.93  0.97  0.94  0.97  0.94

------------------------------------------------------
Validation on the leaving out well CHURCHMAN BIBLE

Model Report
-Accuracy: 0.938119
-Adjacent Accuracy: 0.980198
-F1 Score: 0.937107

Confusion Matrix Results
     Pred    SS  CSiS  FSiS  SiSh    MS    WS     D    PS    BS Total
     True
       SS     8                                                     8
     CSiS          54     1                 1                      56
     FSiS           1    50                                        51
     SiSh                      12                       1          13
       MS                 1          23     5           1          30
       WS                                  85           2          87
        D           1                       1    27     2     3    34
       PS                                   5          70          75
       BS                                                    50    50

Precision  1.00  0.96  0.96  1.00  1.00  0.88  1.00  0.92  0.94  0.94
   Recall  1.00  0.96  0.98  0.92  0.77  0.98  0.79  0.93  1.00  0.94
       F1  1.00  0.96  0.97  0.96  0.87  0.92  0.89  0.93  0.97  0.94

------------------------------------------------------
Final Results
-Average F1 Score: 0.950283

In [ ]:

Use final model to predict the given test data set


In [18]:
# Load test data
test_data = pd.read_csv('validation_data_nofacies.csv')
test_data['Well Name'] = test_data['Well Name'].astype('category')
X_test = test_data.drop(['Formation', 'Well Name', 'Depth'], axis=1)
# Predict facies of unclassified data
Y_predicted = model_final.predict(X_test)
test_data['Facies'] = Y_predicted + 1
# Store the prediction
test_data.to_csv('Prediction3.csv')

In [19]:
test_data


Out[19]:
Formation Well Name Depth GR ILD_log10 DeltaPHI PHIND PE NM_M RELPOS Facies
0 A1 SH STUART 2808.0 66.276 0.630 3.300 10.650 3.591 1 1.000 2
1 A1 SH STUART 2808.5 77.252 0.585 6.500 11.950 3.341 1 0.978 3
2 A1 SH STUART 2809.0 82.899 0.566 9.400 13.600 3.064 1 0.956 2
3 A1 SH STUART 2809.5 80.671 0.593 9.500 13.250 2.977 1 0.933 2
4 A1 SH STUART 2810.0 75.971 0.638 8.700 12.350 3.020 1 0.911 2
5 A1 SH STUART 2810.5 73.955 0.667 6.900 12.250 3.086 1 0.889 3
6 A1 SH STUART 2811.0 77.962 0.674 6.500 12.450 3.092 1 0.867 2
7 A1 SH STUART 2811.5 83.894 0.667 6.300 12.650 3.123 1 0.844 2
8 A1 SH STUART 2812.0 84.424 0.653 6.700 13.050 3.121 1 0.822 2
9 A1 SH STUART 2812.5 83.160 0.642 7.300 12.950 3.127 1 0.800 2
10 A1 SH STUART 2813.0 79.063 0.651 7.300 12.050 3.147 1 0.778 2
11 A1 SH STUART 2813.5 69.002 0.677 6.200 10.800 3.096 1 0.756 2
12 A1 SH STUART 2814.0 63.983 0.690 4.400 9.700 3.103 1 0.733 2
13 A1 SH STUART 2814.5 61.797 0.675 3.500 9.150 3.101 1 0.711 2
14 A1 SH STUART 2815.0 61.372 0.646 2.800 9.300 3.065 1 0.689 2
15 A1 SH STUART 2815.5 63.535 0.621 2.800 9.800 2.982 1 0.667 2
16 A1 SH STUART 2816.0 65.126 0.600 3.300 10.550 2.914 1 0.644 2
17 A1 SH STUART 2816.5 75.930 0.576 3.400 11.900 2.845 1 0.600 2
18 A1 SH STUART 2817.0 85.077 0.584 4.400 12.900 2.854 1 0.578 2
19 A1 SH STUART 2817.5 89.459 0.598 6.600 13.500 2.986 1 0.556 2
20 A1 SH STUART 2818.0 88.619 0.610 7.200 14.800 2.988 1 0.533 2
21 A1 SH STUART 2818.5 81.593 0.636 6.400 13.900 2.998 1 0.511 2
22 A1 SH STUART 2819.0 66.595 0.702 2.800 11.400 2.988 1 0.489 2
23 A1 SH STUART 2819.5 55.081 0.789 2.700 8.150 3.028 1 0.467 2
24 A1 SH STUART 2820.0 48.112 0.840 1.000 7.500 3.073 1 0.444 2
25 A1 SH STUART 2820.5 43.730 0.846 0.400 7.100 3.146 1 0.422 1
26 A1 SH STUART 2821.0 44.097 0.840 0.700 6.650 3.205 1 0.400 1
27 A1 SH STUART 2821.5 46.839 0.842 0.800 6.600 3.254 1 0.378 1
28 A1 SH STUART 2822.0 50.348 0.843 1.100 6.750 3.230 1 0.356 2
29 A1 SH STUART 2822.5 57.129 0.822 2.200 7.300 3.237 1 0.333 2
... ... ... ... ... ... ... ... ... ... ... ...
800 B5 LM CRAWFORD 3146.0 167.803 -0.219 4.270 23.370 3.810 2 0.190 7
801 B5 LM CRAWFORD 3146.5 151.183 -0.057 0.925 17.125 4.153 2 0.172 8
802 B5 LM CRAWFORD 3147.0 123.264 0.067 0.285 14.215 4.404 2 0.155 8
803 B5 LM CRAWFORD 3147.5 108.569 0.234 0.705 12.225 4.499 2 0.138 8
804 B5 LM CRAWFORD 3148.0 101.072 0.427 1.150 10.760 4.392 2 0.121 8
805 B5 LM CRAWFORD 3148.5 91.748 0.625 1.135 9.605 4.254 2 0.103 8
806 B5 LM CRAWFORD 3149.0 83.794 0.749 2.075 7.845 4.023 2 0.086 8
807 B5 LM CRAWFORD 3149.5 83.794 0.749 2.075 7.845 4.023 2 0.086 8
808 B5 LM CRAWFORD 3150.0 79.722 0.771 2.890 6.640 4.040 2 0.069 6
809 B5 LM CRAWFORD 3150.5 76.334 0.800 2.960 6.290 3.997 2 0.052 8
810 B5 LM CRAWFORD 3151.0 73.631 0.800 2.680 6.690 3.828 2 0.034 8
811 B5 LM CRAWFORD 3151.5 76.865 0.772 2.420 8.600 3.535 2 0.017 8
812 C SH CRAWFORD 3152.0 79.924 0.752 2.620 11.510 3.148 1 1.000 2
813 C SH CRAWFORD 3152.5 82.199 0.728 3.725 14.555 2.964 1 0.972 3
814 C SH CRAWFORD 3153.0 79.953 0.700 5.610 16.930 2.793 1 0.944 3
815 C SH CRAWFORD 3153.5 75.881 0.673 6.300 17.570 2.969 1 0.917 3
816 C SH CRAWFORD 3154.0 67.470 0.652 4.775 15.795 3.282 1 0.889 3
817 C SH CRAWFORD 3154.5 58.832 0.640 4.315 13.575 3.642 1 0.861 2
818 C SH CRAWFORD 3155.0 57.946 0.631 3.595 11.305 3.893 1 0.833 2
819 C SH CRAWFORD 3155.5 65.755 0.625 3.465 10.355 3.911 1 0.806 2
820 C SH CRAWFORD 3156.0 69.445 0.617 3.390 11.540 3.820 1 0.778 2
821 C SH CRAWFORD 3156.5 73.389 0.608 3.625 12.775 3.620 1 0.750 2
822 C SH CRAWFORD 3157.0 77.115 0.605 4.140 13.420 3.467 1 0.722 2
823 C SH CRAWFORD 3157.5 79.840 0.596 4.875 13.825 3.360 1 0.694 2
824 C SH CRAWFORD 3158.0 82.616 0.577 5.235 14.845 3.207 1 0.667 2
825 C SH CRAWFORD 3158.5 86.078 0.554 5.040 16.150 3.161 1 0.639 2
826 C SH CRAWFORD 3159.0 88.855 0.539 5.560 16.750 3.118 1 0.611 3
827 C SH CRAWFORD 3159.5 90.490 0.530 6.360 16.780 3.168 1 0.583 3
828 C SH CRAWFORD 3160.0 90.975 0.522 7.035 16.995 3.154 1 0.556 3
829 C SH CRAWFORD 3160.5 90.108 0.513 7.505 17.595 3.125 1 0.528 3

830 rows × 11 columns

Future work, make more customerized objective function. Also, we could use RandomizedSearchCV instead of GridSearchCV to avoild potential local minimal trap and further improve the test results.