notebook.community

Edit and run

Facies classification using Machine Learning

LA Team Submission 6

Lukas Mosser, Alfredo De la Fuente

In this approach for solving the facies classfication problem ( https://github.com/seg/2016-ml-contest. ) we will explore the following statregies:

  • Features Exploration: based on Paolo Bestagini's work, we will consider imputation, normalization and augmentation routines for the initial features.
  • Model tuning: we use TPOT to come up with a good enough pipeline, and then tune the hyperparameters of the model obtained using HYPEROPT.

Packages and Libraries





In [1]:



    


%%sh
pip install pandas
pip install scikit-learn
pip install tpot
pip install hyperopt
pip install xgboost



    


















    






Requirement already satisfied (use --upgrade to upgrade): pandas in /home/alfredo/anaconda2/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /home/alfredo/anaconda2/lib/python2.7/site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): pytz>=2011k in /home/alfredo/anaconda2/lib/python2.7/site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.0 in /home/alfredo/anaconda2/lib/python2.7/site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5 in /home/alfredo/anaconda2/lib/python2.7/site-packages (from python-dateutil->pandas)
Requirement already satisfied (use --upgrade to upgrade): scikit-learn in /home/alfredo/anaconda2/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): tpot in /home/alfredo/anaconda2/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): deap in /home/alfredo/anaconda2/lib/python2.7/site-packages (from tpot)
Requirement already satisfied (use --upgrade to upgrade): numpy in /home/alfredo/anaconda2/lib/python2.7/site-packages (from tpot)
Requirement already satisfied (use --upgrade to upgrade): update-checker in /home/alfredo/anaconda2/lib/python2.7/site-packages (from tpot)
Requirement already satisfied (use --upgrade to upgrade): scipy in /home/alfredo/anaconda2/lib/python2.7/site-packages (from tpot)
Requirement already satisfied (use --upgrade to upgrade): tqdm in /home/alfredo/anaconda2/lib/python2.7/site-packages (from tpot)
Requirement already satisfied (use --upgrade to upgrade): scikit-learn in /home/alfredo/anaconda2/lib/python2.7/site-packages (from tpot)
Requirement already satisfied (use --upgrade to upgrade): requests>=2.3.0 in /home/alfredo/anaconda2/lib/python2.7/site-packages (from update-checker->tpot)
Requirement already satisfied (use --upgrade to upgrade): hyperopt in /home/alfredo/anaconda2/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): numpy in /home/alfredo/anaconda2/lib/python2.7/site-packages (from hyperopt)
Requirement already satisfied (use --upgrade to upgrade): networkx in /home/alfredo/anaconda2/lib/python2.7/site-packages (from hyperopt)
Requirement already satisfied (use --upgrade to upgrade): nose in /home/alfredo/anaconda2/lib/python2.7/site-packages (from hyperopt)
Requirement already satisfied (use --upgrade to upgrade): future in /home/alfredo/anaconda2/lib/python2.7/site-packages (from hyperopt)
Requirement already satisfied (use --upgrade to upgrade): scipy in /home/alfredo/anaconda2/lib/python2.7/site-packages (from hyperopt)
Requirement already satisfied (use --upgrade to upgrade): six in /home/alfredo/anaconda2/lib/python2.7/site-packages (from hyperopt)
Requirement already satisfied (use --upgrade to upgrade): pymongo in /home/alfredo/anaconda2/lib/python2.7/site-packages (from hyperopt)
Requirement already satisfied (use --upgrade to upgrade): decorator>=3.4.0 in /home/alfredo/anaconda2/lib/python2.7/site-packages (from networkx->hyperopt)
Requirement already satisfied (use --upgrade to upgrade): xgboost in /home/alfredo/anaconda2/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): numpy in /home/alfredo/anaconda2/lib/python2.7/site-packages (from xgboost)
Requirement already satisfied (use --upgrade to upgrade): scipy in /home/alfredo/anaconda2/lib/python2.7/site-packages (from xgboost)
Requirement already satisfied (use --upgrade to upgrade): scikit-learn in /home/alfredo/anaconda2/lib/python2.7/site-packages (from xgboost)










    






You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

















In [2]:



    


from __future__ import print_function
import numpy as np
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold , StratifiedKFold
from classification_utilities import display_cm, display_adj_cm
from sklearn.metrics import confusion_matrix, f1_score
from sklearn import preprocessing
from sklearn.model_selection import LeavePGroupsOut
from sklearn.multiclass import OneVsOneClassifier
from scipy.signal import medfilt

Data Preprocessing

We procceed to run Paolo Bestagini's routine to include a small window of values to acount for the spatial component in the log analysis, as well as the gradient information with respect to depth. This will be our prepared training dataset.





In [3]:



    


#Load Data
data = pd.read_csv('../facies_vectors.csv')

# Parameters
feature_names = ['GR', 'ILD_log10', 'DeltaPHI', 'PHIND', 'PE', 'NM_M', 'RELPOS']
facies_names = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS', 'WS', 'D', 'PS', 'BS']
facies_colors = ['#F4D03F', '#F5B041','#DC7633','#6E2C00', '#1B4F72','#2E86C1', '#AED6F1', '#A569BD', '#196F3D']

# Store features and labels
X = data[feature_names].values 
y = data['Facies'].values 

# Store well labels and depths
well = data['Well Name'].values
depth = data['Depth'].values

# Fill 'PE' missing values with mean
imp = preprocessing.Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit(X)
X = imp.transform(X)



    











In [4]:



    


# Feature windows concatenation function
def augment_features_window(X, N_neig):
    
    # Parameters
    N_row = X.shape[0]
    N_feat = X.shape[1]

    # Zero padding
    X = np.vstack((np.zeros((N_neig, N_feat)), X, (np.zeros((N_neig, N_feat)))))

    # Loop over windows
    X_aug = np.zeros((N_row, N_feat*(2*N_neig+1)))
    for r in np.arange(N_row)+N_neig:
        this_row = []
        for c in np.arange(-N_neig,N_neig+1):
            this_row = np.hstack((this_row, X[r+c]))
        X_aug[r-N_neig] = this_row

    return X_aug


# Feature gradient computation function
def augment_features_gradient(X, depth):
    
    # Compute features gradient
    d_diff = np.diff(depth).reshape((-1, 1))
    d_diff[d_diff==0] = 0.001
    X_diff = np.diff(X, axis=0)
    X_grad = X_diff / d_diff
        
    # Compensate for last missing value
    X_grad = np.concatenate((X_grad, np.zeros((1, X_grad.shape[1]))))
    
    return X_grad


# Feature augmentation function
def augment_features(X, well, depth, N_neig=1):
    
    # Augment features
    X_aug = np.zeros((X.shape[0], X.shape[1]*(N_neig*2+2)))
    for w in np.unique(well):
        w_idx = np.where(well == w)[0]
        X_aug_win = augment_features_window(X[w_idx, :], N_neig)
        X_aug_grad = augment_features_gradient(X[w_idx, :], depth[w_idx])
        X_aug[w_idx, :] = np.concatenate((X_aug_win, X_aug_grad), axis=1)
    
    # Find padded rows
    padded_rows = np.unique(np.where(X_aug[:, 0:7] == np.zeros((1, 7)))[0])
    
    return X_aug, padded_rows

X_aug, padded_rows = augment_features(X, well, depth)



    











In [5]:



    


# Initialize model selection methods
lpgo = LeavePGroupsOut(2)

# Generate splits
split_list = []
for train, val in lpgo.split(X, y, groups=data['Well Name']):
    hist_tr = np.histogram(y[train], bins=np.arange(len(facies_names)+1)+.5)
    hist_val = np.histogram(y[val], bins=np.arange(len(facies_names)+1)+.5)
    if np.all(hist_tr[0] != 0) & np.all(hist_val[0] != 0):
        split_list.append({'train':train, 'val':val})
    
        
def preprocess():
    
    # Preprocess data to use in model
    X_train_aux = []
    X_test_aux = []
    y_train_aux = []
    y_test_aux = []
    
    # For each data split
    for split in split_list:
        # Remove padded rows
        split_train_no_pad = np.setdiff1d(split['train'], padded_rows)

        # Select training and validation data from current split
        X_tr = X_aug[split_train_no_pad, :]
        X_v = X_aug[split['val'], :]
        y_tr = y[split_train_no_pad]
        y_v = y[split['val']]

        # Select well labels for validation data
        well_v = well[split['val']]

        # Feature normalization
        scaler = preprocessing.RobustScaler(quantile_range=(25.0, 75.0)).fit(X_tr)
        X_tr = scaler.transform(X_tr)
        X_v = scaler.transform(X_v)

        X_train_aux.append( X_tr )
        X_test_aux.append( X_v )
        y_train_aux.append( y_tr )
        y_test_aux.append (  y_v )

        X_train = np.concatenate( X_train_aux )
        X_test = np.concatenate ( X_test_aux )
        y_train = np.concatenate ( y_train_aux )
        y_test = np.concatenate ( y_test_aux )
    
    return X_train , X_test , y_train , y_test 

X_train, X_test, y_train, y_test = preprocess()
y_train = y_train - 1 
y_test = y_test - 1

Data Analysis

In this section we will run a Cross Validation routine





In [8]:



    


import xgboost as xgb
from xgboost.sklearn import  XGBClassifier
from hyperopt import STATUS_OK, Trials, fmin, hp, tpe
from sklearn.pipeline import make_pipeline



    











In [48]:



    


SEED = 314159265
VALID_SIZE = 0.2
TARGET = 'outcome'

# Scoring and optimization functions

def score(params):
    print("Training with params: ")
    print(params)
    #clf = xgb.XGBClassifier(**params) 
    #clf.fit(X_train, y_train)
    #y_predictions = clf.predict(X_test)
    num_round = int(params['n_estimators'])
    del params['n_estimators']
    dtrain = xgb.DMatrix(X_train, label=y_train)
    dvalid = xgb.DMatrix(X_test, label=y_test)
    watchlist = [(dvalid, 'eval'), (dtrain, 'train')]
    gbm_model = xgb.train(params, dtrain, num_round,
                          evals=watchlist,
                          verbose_eval=True)
    y_predictions = gbm_model.predict(dvalid,
                                    ntree_limit=gbm_model.best_iteration + 1)
    
    score = f1_score (y_test, y_predictions , average ='micro')
    print("\tScore {0}\n\n".format(score))
    loss = 1 - score
    return {'loss': loss, 'status': STATUS_OK}


def optimize(random_state=SEED):
    space = {
        'n_estimators': hp.quniform('n_estimators', 100, 150, 1),
        'eta': hp.quniform('eta', 0.025, 0.5, 0.025),
        'max_depth':  hp.choice('max_depth', np.arange(1, 14, dtype=int)),
        'min_child_weight': hp.quniform('min_child_weight', 1, 6, 1),
        'subsample': hp.quniform('subsample', 0.5, 1, 0.05),
        'gamma': hp.quniform('gamma', 0.5, 1, 0.05),
        'colsample_bytree': hp.quniform('colsample_bytree', 0.5, 1, 0.05),
        'eval_metric': 'mlogloss',
        'objective': 'multi:softmax',
        'nthread': 4,
        'booster': 'gbtree',
        'tree_method': 'exact',
        'silent': 1,
        'num_class' : 9,
        'seed': random_state
    }
    # Use the fmin function from Hyperopt to find the best hyperparameters
    best = fmin(score, space, algo=tpe.suggest,  max_evals=5)
    return best



    











In [49]:



    


best_hyperparams = optimize()
print("The best hyperparameters are: ", "\n")
print(best_hyperparams)



    


















    






Training with params: 
{'colsample_bytree': 0.8500000000000001, 'silent': 1, 'eval_metric': 'mlogloss', 'nthread': 4, 'min_child_weight': 4.0, 'n_estimators': 148.0, 'subsample': 0.9, 'eta': 0.15000000000000002, 'objective': 'multi:softmax', 'num_class': 9, 'tree_method': 'exact', 'seed': 314159265, 'max_depth': 9, 'gamma': 0.75, 'booster': 'gbtree'}
[0]	eval-mlogloss:1.77004	train-mlogloss:1.75484
[1]	eval-mlogloss:1.51325	train-mlogloss:1.48608
[2]	eval-mlogloss:1.31655	train-mlogloss:1.28232
[3]	eval-mlogloss:1.16507	train-mlogloss:1.12506
[4]	eval-mlogloss:1.0393	train-mlogloss:0.995635
[5]	eval-mlogloss:0.935651	train-mlogloss:0.889398
[6]	eval-mlogloss:0.847376	train-mlogloss:0.798796
[7]	eval-mlogloss:0.770582	train-mlogloss:0.720501
[8]	eval-mlogloss:0.703384	train-mlogloss:0.652322
[9]	eval-mlogloss:0.64605	train-mlogloss:0.594215
[10]	eval-mlogloss:0.594241	train-mlogloss:0.541877
[11]	eval-mlogloss:0.548435	train-mlogloss:0.49588
[12]	eval-mlogloss:0.507176	train-mlogloss:0.454374
[13]	eval-mlogloss:0.472222	train-mlogloss:0.419359
[14]	eval-mlogloss:0.440209	train-mlogloss:0.38774
[15]	eval-mlogloss:0.410642	train-mlogloss:0.358263
[16]	eval-mlogloss:0.384208	train-mlogloss:0.332742
[17]	eval-mlogloss:0.360306	train-mlogloss:0.308959
[18]	eval-mlogloss:0.339444	train-mlogloss:0.288337
[19]	eval-mlogloss:0.320635	train-mlogloss:0.27024
[20]	eval-mlogloss:0.30304	train-mlogloss:0.253228
[21]	eval-mlogloss:0.287396	train-mlogloss:0.238097
[22]	eval-mlogloss:0.274009	train-mlogloss:0.225423
[23]	eval-mlogloss:0.260853	train-mlogloss:0.212888
[24]	eval-mlogloss:0.249767	train-mlogloss:0.202529
[25]	eval-mlogloss:0.238788	train-mlogloss:0.192004
[26]	eval-mlogloss:0.22815	train-mlogloss:0.181666
[27]	eval-mlogloss:0.219896	train-mlogloss:0.173735
[28]	eval-mlogloss:0.210887	train-mlogloss:0.165314
[29]	eval-mlogloss:0.203215	train-mlogloss:0.15805
[30]	eval-mlogloss:0.195757	train-mlogloss:0.151125
[31]	eval-mlogloss:0.187306	train-mlogloss:0.143409
[32]	eval-mlogloss:0.179986	train-mlogloss:0.136504
[33]	eval-mlogloss:0.173018	train-mlogloss:0.130126
[34]	eval-mlogloss:0.166742	train-mlogloss:0.124243
[35]	eval-mlogloss:0.160619	train-mlogloss:0.118565
[36]	eval-mlogloss:0.154906	train-mlogloss:0.1134
[37]	eval-mlogloss:0.149827	train-mlogloss:0.108622
[38]	eval-mlogloss:0.144828	train-mlogloss:0.103783
[39]	eval-mlogloss:0.139478	train-mlogloss:0.09876
[40]	eval-mlogloss:0.134523	train-mlogloss:0.094516
[41]	eval-mlogloss:0.13065	train-mlogloss:0.09091
[42]	eval-mlogloss:0.126109	train-mlogloss:0.086819
[43]	eval-mlogloss:0.121928	train-mlogloss:0.082895
[44]	eval-mlogloss:0.118028	train-mlogloss:0.079453
[45]	eval-mlogloss:0.11356	train-mlogloss:0.075485
[46]	eval-mlogloss:0.110264	train-mlogloss:0.072469
[47]	eval-mlogloss:0.107411	train-mlogloss:0.069873
[48]	eval-mlogloss:0.103915	train-mlogloss:0.066762
[49]	eval-mlogloss:0.100752	train-mlogloss:0.063897
[50]	eval-mlogloss:0.097958	train-mlogloss:0.06156
[51]	eval-mlogloss:0.095155	train-mlogloss:0.059015
[52]	eval-mlogloss:0.092131	train-mlogloss:0.056393
[53]	eval-mlogloss:0.089288	train-mlogloss:0.053926
[54]	eval-mlogloss:0.08711	train-mlogloss:0.052081
[55]	eval-mlogloss:0.085152	train-mlogloss:0.050433
[56]	eval-mlogloss:0.082962	train-mlogloss:0.048555
[57]	eval-mlogloss:0.080881	train-mlogloss:0.046698
[58]	eval-mlogloss:0.078692	train-mlogloss:0.044856
[59]	eval-mlogloss:0.076667	train-mlogloss:0.043168
[60]	eval-mlogloss:0.074782	train-mlogloss:0.041527
[61]	eval-mlogloss:0.073075	train-mlogloss:0.040189
[62]	eval-mlogloss:0.071012	train-mlogloss:0.038506
[63]	eval-mlogloss:0.069498	train-mlogloss:0.037268
[64]	eval-mlogloss:0.067669	train-mlogloss:0.035846
[65]	eval-mlogloss:0.066142	train-mlogloss:0.03463
[66]	eval-mlogloss:0.06487	train-mlogloss:0.033471
[67]	eval-mlogloss:0.063373	train-mlogloss:0.0323
[68]	eval-mlogloss:0.061974	train-mlogloss:0.031185
[69]	eval-mlogloss:0.060628	train-mlogloss:0.030123
[70]	eval-mlogloss:0.059831	train-mlogloss:0.029479
[71]	eval-mlogloss:0.058782	train-mlogloss:0.028643
[72]	eval-mlogloss:0.057335	train-mlogloss:0.027521
[73]	eval-mlogloss:0.055983	train-mlogloss:0.026461
[74]	eval-mlogloss:0.054857	train-mlogloss:0.025644
[75]	eval-mlogloss:0.053828	train-mlogloss:0.024907
[76]	eval-mlogloss:0.052583	train-mlogloss:0.023964
[77]	eval-mlogloss:0.051601	train-mlogloss:0.023283
[78]	eval-mlogloss:0.050518	train-mlogloss:0.022443
[79]	eval-mlogloss:0.049933	train-mlogloss:0.021966
[80]	eval-mlogloss:0.049042	train-mlogloss:0.021303
[81]	eval-mlogloss:0.048475	train-mlogloss:0.020866
[82]	eval-mlogloss:0.047539	train-mlogloss:0.020203
[83]	eval-mlogloss:0.046641	train-mlogloss:0.019567
[84]	eval-mlogloss:0.04593	train-mlogloss:0.019053
[85]	eval-mlogloss:0.045267	train-mlogloss:0.018558
[86]	eval-mlogloss:0.044716	train-mlogloss:0.018166
[87]	eval-mlogloss:0.044155	train-mlogloss:0.017757
[88]	eval-mlogloss:0.043403	train-mlogloss:0.01719
[89]	eval-mlogloss:0.042785	train-mlogloss:0.01672
[90]	eval-mlogloss:0.04229	train-mlogloss:0.016347
[91]	eval-mlogloss:0.041787	train-mlogloss:0.016025
[92]	eval-mlogloss:0.041198	train-mlogloss:0.015666
[93]	eval-mlogloss:0.040804	train-mlogloss:0.015406
[94]	eval-mlogloss:0.040351	train-mlogloss:0.015118
[95]	eval-mlogloss:0.039917	train-mlogloss:0.014818
[96]	eval-mlogloss:0.039725	train-mlogloss:0.014635
[97]	eval-mlogloss:0.039294	train-mlogloss:0.014343
[98]	eval-mlogloss:0.039041	train-mlogloss:0.014163
[99]	eval-mlogloss:0.038654	train-mlogloss:0.013873
[100]	eval-mlogloss:0.038336	train-mlogloss:0.013704
[101]	eval-mlogloss:0.038138	train-mlogloss:0.013552
[102]	eval-mlogloss:0.037895	train-mlogloss:0.01338
[103]	eval-mlogloss:0.037624	train-mlogloss:0.013187
[104]	eval-mlogloss:0.037433	train-mlogloss:0.013077
[105]	eval-mlogloss:0.037225	train-mlogloss:0.01294
[106]	eval-mlogloss:0.036978	train-mlogloss:0.012761
[107]	eval-mlogloss:0.03666	train-mlogloss:0.01254
[108]	eval-mlogloss:0.036486	train-mlogloss:0.012426
[109]	eval-mlogloss:0.036242	train-mlogloss:0.012227
[110]	eval-mlogloss:0.036007	train-mlogloss:0.012133
[111]	eval-mlogloss:0.035735	train-mlogloss:0.011947
[112]	eval-mlogloss:0.035608	train-mlogloss:0.011853
[113]	eval-mlogloss:0.035451	train-mlogloss:0.011744
[114]	eval-mlogloss:0.035227	train-mlogloss:0.011612
[115]	eval-mlogloss:0.035013	train-mlogloss:0.011487
[116]	eval-mlogloss:0.034848	train-mlogloss:0.01136
[117]	eval-mlogloss:0.034715	train-mlogloss:0.011269
[118]	eval-mlogloss:0.034597	train-mlogloss:0.011159
[119]	eval-mlogloss:0.034431	train-mlogloss:0.011081
[120]	eval-mlogloss:0.03425	train-mlogloss:0.011006
[121]	eval-mlogloss:0.03401	train-mlogloss:0.010901
[122]	eval-mlogloss:0.033744	train-mlogloss:0.01078
[123]	eval-mlogloss:0.033637	train-mlogloss:0.010712
[124]	eval-mlogloss:0.033449	train-mlogloss:0.010621
[125]	eval-mlogloss:0.033386	train-mlogloss:0.010585
[126]	eval-mlogloss:0.03331	train-mlogloss:0.010549
[127]	eval-mlogloss:0.033202	train-mlogloss:0.010483
[128]	eval-mlogloss:0.033064	train-mlogloss:0.010427
[129]	eval-mlogloss:0.03295	train-mlogloss:0.010363
[130]	eval-mlogloss:0.032851	train-mlogloss:0.010292
[131]	eval-mlogloss:0.032744	train-mlogloss:0.010225
[132]	eval-mlogloss:0.03264	train-mlogloss:0.010157
[133]	eval-mlogloss:0.032553	train-mlogloss:0.010103
[134]	eval-mlogloss:0.032441	train-mlogloss:0.010039
[135]	eval-mlogloss:0.032316	train-mlogloss:0.009974
[136]	eval-mlogloss:0.032236	train-mlogloss:0.009931
[137]	eval-mlogloss:0.032182	train-mlogloss:0.009897
[138]	eval-mlogloss:0.032111	train-mlogloss:0.009863
[139]	eval-mlogloss:0.032051	train-mlogloss:0.009826
[140]	eval-mlogloss:0.031924	train-mlogloss:0.009761
[141]	eval-mlogloss:0.031845	train-mlogloss:0.009709
[142]	eval-mlogloss:0.031726	train-mlogloss:0.009656
[143]	eval-mlogloss:0.031667	train-mlogloss:0.009621
[144]	eval-mlogloss:0.031598	train-mlogloss:0.009565
[145]	eval-mlogloss:0.031484	train-mlogloss:0.00949
[146]	eval-mlogloss:0.031468	train-mlogloss:0.009471
[147]	eval-mlogloss:0.031395	train-mlogloss:0.00945
	Score 0.994896053778


Training with params: 
{'colsample_bytree': 0.9500000000000001, 'silent': 1, 'eval_metric': 'mlogloss', 'nthread': 4, 'min_child_weight': 3.0, 'n_estimators': 130.0, 'subsample': 0.5, 'eta': 0.125, 'objective': 'multi:softmax', 'num_class': 9, 'tree_method': 'exact', 'seed': 314159265, 'max_depth': 6, 'gamma': 0.5, 'booster': 'gbtree'}
[0]	eval-mlogloss:1.92789	train-mlogloss:1.92113
[1]	eval-mlogloss:1.74143	train-mlogloss:1.72934
[2]	eval-mlogloss:1.59495	train-mlogloss:1.57852
[3]	eval-mlogloss:1.47371	train-mlogloss:1.45474
[4]	eval-mlogloss:1.37346	train-mlogloss:1.35194
[5]	eval-mlogloss:1.28305	train-mlogloss:1.25967
[6]	eval-mlogloss:1.20614	train-mlogloss:1.18007
[7]	eval-mlogloss:1.13975	train-mlogloss:1.11183
[8]	eval-mlogloss:1.0807	train-mlogloss:1.05057
[9]	eval-mlogloss:1.02624	train-mlogloss:0.994854
[10]	eval-mlogloss:0.978985	train-mlogloss:0.946839
[11]	eval-mlogloss:0.932916	train-mlogloss:0.900608
[12]	eval-mlogloss:0.892368	train-mlogloss:0.859662
[13]	eval-mlogloss:0.85531	train-mlogloss:0.822238
[14]	eval-mlogloss:0.823397	train-mlogloss:0.789734
[15]	eval-mlogloss:0.79303	train-mlogloss:0.75955
[16]	eval-mlogloss:0.765125	train-mlogloss:0.731456
[17]	eval-mlogloss:0.740175	train-mlogloss:0.706487
[18]	eval-mlogloss:0.717183	train-mlogloss:0.68345
[19]	eval-mlogloss:0.695516	train-mlogloss:0.66164
[20]	eval-mlogloss:0.674492	train-mlogloss:0.640569
[21]	eval-mlogloss:0.655986	train-mlogloss:0.622641
[22]	eval-mlogloss:0.638493	train-mlogloss:0.605001
[23]	eval-mlogloss:0.622749	train-mlogloss:0.588831
[24]	eval-mlogloss:0.606864	train-mlogloss:0.5734
[25]	eval-mlogloss:0.59366	train-mlogloss:0.560426
[26]	eval-mlogloss:0.579328	train-mlogloss:0.545958
[27]	eval-mlogloss:0.566982	train-mlogloss:0.533347
[28]	eval-mlogloss:0.554634	train-mlogloss:0.521389
[29]	eval-mlogloss:0.543321	train-mlogloss:0.510202
[30]	eval-mlogloss:0.532859	train-mlogloss:0.499955
[31]	eval-mlogloss:0.521898	train-mlogloss:0.489097
[32]	eval-mlogloss:0.511452	train-mlogloss:0.478537
[33]	eval-mlogloss:0.501348	train-mlogloss:0.468414
[34]	eval-mlogloss:0.492825	train-mlogloss:0.459942
[35]	eval-mlogloss:0.483348	train-mlogloss:0.450887
[36]	eval-mlogloss:0.475146	train-mlogloss:0.44293
[37]	eval-mlogloss:0.466525	train-mlogloss:0.434177
[38]	eval-mlogloss:0.458673	train-mlogloss:0.426505
[39]	eval-mlogloss:0.451264	train-mlogloss:0.419337
[40]	eval-mlogloss:0.442097	train-mlogloss:0.410212
[41]	eval-mlogloss:0.435349	train-mlogloss:0.403442
[42]	eval-mlogloss:0.428525	train-mlogloss:0.396795
[43]	eval-mlogloss:0.421748	train-mlogloss:0.39012
[44]	eval-mlogloss:0.414968	train-mlogloss:0.383047
[45]	eval-mlogloss:0.408866	train-mlogloss:0.376949
[46]	eval-mlogloss:0.403123	train-mlogloss:0.371206
[47]	eval-mlogloss:0.396697	train-mlogloss:0.36499
[48]	eval-mlogloss:0.390728	train-mlogloss:0.358826
[49]	eval-mlogloss:0.384709	train-mlogloss:0.35266
[50]	eval-mlogloss:0.380819	train-mlogloss:0.348798
[51]	eval-mlogloss:0.374608	train-mlogloss:0.342568
[52]	eval-mlogloss:0.369633	train-mlogloss:0.337643
[53]	eval-mlogloss:0.363941	train-mlogloss:0.331854
[54]	eval-mlogloss:0.358222	train-mlogloss:0.326021
[55]	eval-mlogloss:0.352024	train-mlogloss:0.319775
[56]	eval-mlogloss:0.346404	train-mlogloss:0.313722
[57]	eval-mlogloss:0.34143	train-mlogloss:0.308719
[58]	eval-mlogloss:0.336872	train-mlogloss:0.304148
[59]	eval-mlogloss:0.330147	train-mlogloss:0.297439
[60]	eval-mlogloss:0.324959	train-mlogloss:0.292178
[61]	eval-mlogloss:0.320224	train-mlogloss:0.287564
[62]	eval-mlogloss:0.314774	train-mlogloss:0.282033
[63]	eval-mlogloss:0.309622	train-mlogloss:0.277189
[64]	eval-mlogloss:0.304038	train-mlogloss:0.271473
[65]	eval-mlogloss:0.299212	train-mlogloss:0.266632
[66]	eval-mlogloss:0.294991	train-mlogloss:0.262218
[67]	eval-mlogloss:0.291009	train-mlogloss:0.258173
[68]	eval-mlogloss:0.286676	train-mlogloss:0.253913
[69]	eval-mlogloss:0.282276	train-mlogloss:0.249318
[70]	eval-mlogloss:0.277473	train-mlogloss:0.244461
[71]	eval-mlogloss:0.273733	train-mlogloss:0.240793
[72]	eval-mlogloss:0.270136	train-mlogloss:0.237315
[73]	eval-mlogloss:0.266618	train-mlogloss:0.233829
[74]	eval-mlogloss:0.262503	train-mlogloss:0.22939
[75]	eval-mlogloss:0.259873	train-mlogloss:0.226716
[76]	eval-mlogloss:0.254013	train-mlogloss:0.221095
[77]	eval-mlogloss:0.249635	train-mlogloss:0.216514
[78]	eval-mlogloss:0.245839	train-mlogloss:0.2126
[79]	eval-mlogloss:0.242239	train-mlogloss:0.208699
[80]	eval-mlogloss:0.237601	train-mlogloss:0.203929
[81]	eval-mlogloss:0.234135	train-mlogloss:0.200337
[82]	eval-mlogloss:0.229477	train-mlogloss:0.195704
[83]	eval-mlogloss:0.226223	train-mlogloss:0.19273
[84]	eval-mlogloss:0.223124	train-mlogloss:0.189368
[85]	eval-mlogloss:0.218618	train-mlogloss:0.184952
[86]	eval-mlogloss:0.214466	train-mlogloss:0.180591
[87]	eval-mlogloss:0.210619	train-mlogloss:0.176713
[88]	eval-mlogloss:0.206594	train-mlogloss:0.172659
[89]	eval-mlogloss:0.202847	train-mlogloss:0.169023
[90]	eval-mlogloss:0.199469	train-mlogloss:0.165527
[91]	eval-mlogloss:0.195479	train-mlogloss:0.161528
[92]	eval-mlogloss:0.192669	train-mlogloss:0.1586
[93]	eval-mlogloss:0.190107	train-mlogloss:0.15612
[94]	eval-mlogloss:0.186383	train-mlogloss:0.152612
[95]	eval-mlogloss:0.183059	train-mlogloss:0.149182
[96]	eval-mlogloss:0.179578	train-mlogloss:0.145782
[97]	eval-mlogloss:0.177031	train-mlogloss:0.143446
[98]	eval-mlogloss:0.174724	train-mlogloss:0.141107
[99]	eval-mlogloss:0.172259	train-mlogloss:0.138671
[100]	eval-mlogloss:0.170095	train-mlogloss:0.136556
[101]	eval-mlogloss:0.167157	train-mlogloss:0.133591
[102]	eval-mlogloss:0.164985	train-mlogloss:0.131501
[103]	eval-mlogloss:0.162551	train-mlogloss:0.129025
[104]	eval-mlogloss:0.160533	train-mlogloss:0.127046
[105]	eval-mlogloss:0.158103	train-mlogloss:0.124521
[106]	eval-mlogloss:0.155907	train-mlogloss:0.122441
[107]	eval-mlogloss:0.15401	train-mlogloss:0.120553
[108]	eval-mlogloss:0.151612	train-mlogloss:0.118308
[109]	eval-mlogloss:0.149458	train-mlogloss:0.116192
[110]	eval-mlogloss:0.14762	train-mlogloss:0.11443
[111]	eval-mlogloss:0.145525	train-mlogloss:0.112536
[112]	eval-mlogloss:0.143527	train-mlogloss:0.110399
[113]	eval-mlogloss:0.141293	train-mlogloss:0.108321
[114]	eval-mlogloss:0.139064	train-mlogloss:0.106285
[115]	eval-mlogloss:0.137502	train-mlogloss:0.104795
[116]	eval-mlogloss:0.135344	train-mlogloss:0.102843
[117]	eval-mlogloss:0.13299	train-mlogloss:0.10063
[118]	eval-mlogloss:0.131504	train-mlogloss:0.099218
[119]	eval-mlogloss:0.129841	train-mlogloss:0.097641
[120]	eval-mlogloss:0.128114	train-mlogloss:0.096126
[121]	eval-mlogloss:0.126309	train-mlogloss:0.09441
[122]	eval-mlogloss:0.124784	train-mlogloss:0.092931
[123]	eval-mlogloss:0.122924	train-mlogloss:0.091358
[124]	eval-mlogloss:0.120719	train-mlogloss:0.089282
[125]	eval-mlogloss:0.118647	train-mlogloss:0.087392
[126]	eval-mlogloss:0.117512	train-mlogloss:0.086096
[127]	eval-mlogloss:0.115977	train-mlogloss:0.08465
[128]	eval-mlogloss:0.114131	train-mlogloss:0.082792
[129]	eval-mlogloss:0.112955	train-mlogloss:0.081714
	Score 0.985850035271


Training with params: 
{'colsample_bytree': 0.55, 'silent': 1, 'eval_metric': 'mlogloss', 'nthread': 4, 'min_child_weight': 2.0, 'n_estimators': 117.0, 'subsample': 0.9500000000000001, 'eta': 0.42500000000000004, 'objective': 'multi:softmax', 'num_class': 9, 'tree_method': 'exact', 'seed': 314159265, 'max_depth': 6, 'gamma': 0.75, 'booster': 'gbtree'}
[0]	eval-mlogloss:1.44469	train-mlogloss:1.41586
[1]	eval-mlogloss:1.17193	train-mlogloss:1.13522
[2]	eval-mlogloss:0.986545	train-mlogloss:0.943008
[3]	eval-mlogloss:0.860391	train-mlogloss:0.816106
[4]	eval-mlogloss:0.761685	train-mlogloss:0.716623
[5]	eval-mlogloss:0.684395	train-mlogloss:0.639716
[6]	eval-mlogloss:0.621523	train-mlogloss:0.578735
[7]	eval-mlogloss:0.573709	train-mlogloss:0.530881
[8]	eval-mlogloss:0.53843	train-mlogloss:0.495267
[9]	eval-mlogloss:0.504356	train-mlogloss:0.460889
[10]	eval-mlogloss:0.469825	train-mlogloss:0.426732
[11]	eval-mlogloss:0.443554	train-mlogloss:0.400875
[12]	eval-mlogloss:0.420191	train-mlogloss:0.378074
[13]	eval-mlogloss:0.39945	train-mlogloss:0.357973
[14]	eval-mlogloss:0.378798	train-mlogloss:0.337619
[15]	eval-mlogloss:0.361147	train-mlogloss:0.320762
[16]	eval-mlogloss:0.342794	train-mlogloss:0.301609
[17]	eval-mlogloss:0.323657	train-mlogloss:0.283237
[18]	eval-mlogloss:0.308333	train-mlogloss:0.266414
[19]	eval-mlogloss:0.293466	train-mlogloss:0.251337
[20]	eval-mlogloss:0.281558	train-mlogloss:0.238856
[21]	eval-mlogloss:0.269958	train-mlogloss:0.227889
[22]	eval-mlogloss:0.260351	train-mlogloss:0.218506
[23]	eval-mlogloss:0.251334	train-mlogloss:0.2096
[24]	eval-mlogloss:0.236967	train-mlogloss:0.194475
[25]	eval-mlogloss:0.226366	train-mlogloss:0.182709
[26]	eval-mlogloss:0.217202	train-mlogloss:0.173542
[27]	eval-mlogloss:0.206886	train-mlogloss:0.162952
[28]	eval-mlogloss:0.19607	train-mlogloss:0.152063
[29]	eval-mlogloss:0.187501	train-mlogloss:0.14361
[30]	eval-mlogloss:0.1793	train-mlogloss:0.135822
[31]	eval-mlogloss:0.166798	train-mlogloss:0.124141
[32]	eval-mlogloss:0.159243	train-mlogloss:0.117409
[33]	eval-mlogloss:0.152074	train-mlogloss:0.110558
[34]	eval-mlogloss:0.144023	train-mlogloss:0.102845
[35]	eval-mlogloss:0.13675	train-mlogloss:0.096394
[36]	eval-mlogloss:0.131577	train-mlogloss:0.091318
[37]	eval-mlogloss:0.12553	train-mlogloss:0.085836
[38]	eval-mlogloss:0.119122	train-mlogloss:0.080076
[39]	eval-mlogloss:0.113028	train-mlogloss:0.074467
[40]	eval-mlogloss:0.107772	train-mlogloss:0.070087
[41]	eval-mlogloss:0.102448	train-mlogloss:0.06531
[42]	eval-mlogloss:0.098517	train-mlogloss:0.062047
[43]	eval-mlogloss:0.094599	train-mlogloss:0.058658
[44]	eval-mlogloss:0.09125	train-mlogloss:0.055896
[45]	eval-mlogloss:0.087873	train-mlogloss:0.053087
[46]	eval-mlogloss:0.083506	train-mlogloss:0.04973
[47]	eval-mlogloss:0.081277	train-mlogloss:0.047863
[48]	eval-mlogloss:0.077665	train-mlogloss:0.044746
[49]	eval-mlogloss:0.07537	train-mlogloss:0.04277
[50]	eval-mlogloss:0.071826	train-mlogloss:0.040171
[51]	eval-mlogloss:0.069458	train-mlogloss:0.038443
[52]	eval-mlogloss:0.06684	train-mlogloss:0.036258
[53]	eval-mlogloss:0.065283	train-mlogloss:0.034853
[54]	eval-mlogloss:0.063765	train-mlogloss:0.033463
[55]	eval-mlogloss:0.062168	train-mlogloss:0.032256
[56]	eval-mlogloss:0.060171	train-mlogloss:0.030683
[57]	eval-mlogloss:0.058071	train-mlogloss:0.029274
[58]	eval-mlogloss:0.056495	train-mlogloss:0.028018
[59]	eval-mlogloss:0.054951	train-mlogloss:0.026845
[60]	eval-mlogloss:0.053739	train-mlogloss:0.025963
[61]	eval-mlogloss:0.051921	train-mlogloss:0.02457
[62]	eval-mlogloss:0.05081	train-mlogloss:0.023786
[63]	eval-mlogloss:0.049445	train-mlogloss:0.022786
[64]	eval-mlogloss:0.048847	train-mlogloss:0.02233
[65]	eval-mlogloss:0.047858	train-mlogloss:0.021529
[66]	eval-mlogloss:0.046979	train-mlogloss:0.020879
[67]	eval-mlogloss:0.045875	train-mlogloss:0.020085
[68]	eval-mlogloss:0.044899	train-mlogloss:0.019264
[69]	eval-mlogloss:0.044387	train-mlogloss:0.018948
[70]	eval-mlogloss:0.043987	train-mlogloss:0.018604
[71]	eval-mlogloss:0.04315	train-mlogloss:0.018044
[72]	eval-mlogloss:0.042621	train-mlogloss:0.017658
[73]	eval-mlogloss:0.041861	train-mlogloss:0.01697
[74]	eval-mlogloss:0.041094	train-mlogloss:0.016369
[75]	eval-mlogloss:0.04028	train-mlogloss:0.015707
[76]	eval-mlogloss:0.039965	train-mlogloss:0.015448
[77]	eval-mlogloss:0.039632	train-mlogloss:0.015222
[78]	eval-mlogloss:0.038911	train-mlogloss:0.014781
[79]	eval-mlogloss:0.03842	train-mlogloss:0.014384
[80]	eval-mlogloss:0.03805	train-mlogloss:0.014114
[81]	eval-mlogloss:0.037419	train-mlogloss:0.013699
[82]	eval-mlogloss:0.037129	train-mlogloss:0.013481
[83]	eval-mlogloss:0.036864	train-mlogloss:0.013305
[84]	eval-mlogloss:0.036463	train-mlogloss:0.013133
[85]	eval-mlogloss:0.035883	train-mlogloss:0.012762
[86]	eval-mlogloss:0.035613	train-mlogloss:0.012599
[87]	eval-mlogloss:0.035061	train-mlogloss:0.012273
[88]	eval-mlogloss:0.034732	train-mlogloss:0.012078
[89]	eval-mlogloss:0.034434	train-mlogloss:0.011897
[90]	eval-mlogloss:0.034151	train-mlogloss:0.011678
[91]	eval-mlogloss:0.033896	train-mlogloss:0.011505
[92]	eval-mlogloss:0.033735	train-mlogloss:0.011404
[93]	eval-mlogloss:0.0337	train-mlogloss:0.011325
[94]	eval-mlogloss:0.033574	train-mlogloss:0.011257
[95]	eval-mlogloss:0.03329	train-mlogloss:0.011102
[96]	eval-mlogloss:0.033183	train-mlogloss:0.010994
[97]	eval-mlogloss:0.033044	train-mlogloss:0.010939
[98]	eval-mlogloss:0.032857	train-mlogloss:0.010844
[99]	eval-mlogloss:0.032747	train-mlogloss:0.01078
[100]	eval-mlogloss:0.032654	train-mlogloss:0.010734
[101]	eval-mlogloss:0.032537	train-mlogloss:0.010633
[102]	eval-mlogloss:0.032316	train-mlogloss:0.0105
[103]	eval-mlogloss:0.03222	train-mlogloss:0.010461
[104]	eval-mlogloss:0.03187	train-mlogloss:0.01028
[105]	eval-mlogloss:0.031537	train-mlogloss:0.01012
[106]	eval-mlogloss:0.031483	train-mlogloss:0.010093
[107]	eval-mlogloss:0.031212	train-mlogloss:0.009981
[108]	eval-mlogloss:0.031192	train-mlogloss:0.009946
[109]	eval-mlogloss:0.031041	train-mlogloss:0.009894
[110]	eval-mlogloss:0.031006	train-mlogloss:0.009868
[111]	eval-mlogloss:0.030843	train-mlogloss:0.009784
[112]	eval-mlogloss:0.030838	train-mlogloss:0.009766
[113]	eval-mlogloss:0.030819	train-mlogloss:0.009757
[114]	eval-mlogloss:0.030726	train-mlogloss:0.009704
[115]	eval-mlogloss:0.030594	train-mlogloss:0.009666
[116]	eval-mlogloss:0.03059	train-mlogloss:0.00966
	Score 0.99485455828


Training with params: 
{'colsample_bytree': 0.9, 'silent': 1, 'eval_metric': 'mlogloss', 'nthread': 4, 'min_child_weight': 4.0, 'n_estimators': 122.0, 'subsample': 0.6000000000000001, 'eta': 0.05, 'objective': 'multi:softmax', 'num_class': 9, 'tree_method': 'exact', 'seed': 314159265, 'max_depth': 4, 'gamma': 0.8, 'booster': 'gbtree'}
[0]	eval-mlogloss:2.11124	train-mlogloss:2.11045
[1]	eval-mlogloss:2.03859	train-mlogloss:2.03638
[2]	eval-mlogloss:1.97369	train-mlogloss:1.97042
[3]	eval-mlogloss:1.91449	train-mlogloss:1.90983
[4]	eval-mlogloss:1.86084	train-mlogloss:1.85515
[5]	eval-mlogloss:1.81035	train-mlogloss:1.80414
[6]	eval-mlogloss:1.76298	train-mlogloss:1.75614
[7]	eval-mlogloss:1.72005	train-mlogloss:1.71261
[8]	eval-mlogloss:1.68019	train-mlogloss:1.67196
[9]	eval-mlogloss:1.64229	train-mlogloss:1.63315
[10]	eval-mlogloss:1.60703	train-mlogloss:1.59729
[11]	eval-mlogloss:1.57416	train-mlogloss:1.56353
[12]	eval-mlogloss:1.54277	train-mlogloss:1.53131
[13]	eval-mlogloss:1.51388	train-mlogloss:1.50196
[14]	eval-mlogloss:1.48692	train-mlogloss:1.47406
[15]	eval-mlogloss:1.46083	train-mlogloss:1.44712
[16]	eval-mlogloss:1.43588	train-mlogloss:1.42119
[17]	eval-mlogloss:1.41195	train-mlogloss:1.39633
[18]	eval-mlogloss:1.38915	train-mlogloss:1.37316
[19]	eval-mlogloss:1.36734	train-mlogloss:1.35031
[20]	eval-mlogloss:1.34671	train-mlogloss:1.32883
[21]	eval-mlogloss:1.32691	train-mlogloss:1.30857
[22]	eval-mlogloss:1.30786	train-mlogloss:1.28876
[23]	eval-mlogloss:1.28995	train-mlogloss:1.27013
[24]	eval-mlogloss:1.27235	train-mlogloss:1.25201
[25]	eval-mlogloss:1.25522	train-mlogloss:1.23457
[26]	eval-mlogloss:1.2386	train-mlogloss:1.21778
[27]	eval-mlogloss:1.22228	train-mlogloss:1.20112
[28]	eval-mlogloss:1.20716	train-mlogloss:1.18562
[29]	eval-mlogloss:1.19228	train-mlogloss:1.1708
[30]	eval-mlogloss:1.17748	train-mlogloss:1.1558
[31]	eval-mlogloss:1.16389	train-mlogloss:1.14209
[32]	eval-mlogloss:1.15025	train-mlogloss:1.12808
[33]	eval-mlogloss:1.13757	train-mlogloss:1.11511
[34]	eval-mlogloss:1.12524	train-mlogloss:1.10234
[35]	eval-mlogloss:1.11334	train-mlogloss:1.09022
[36]	eval-mlogloss:1.10234	train-mlogloss:1.07895
[37]	eval-mlogloss:1.09115	train-mlogloss:1.06759
[38]	eval-mlogloss:1.08069	train-mlogloss:1.05708
[39]	eval-mlogloss:1.07011	train-mlogloss:1.04619
[40]	eval-mlogloss:1.05953	train-mlogloss:1.03554
[41]	eval-mlogloss:1.04988	train-mlogloss:1.02569
[42]	eval-mlogloss:1.04056	train-mlogloss:1.0164
[43]	eval-mlogloss:1.03146	train-mlogloss:1.00704
[44]	eval-mlogloss:1.02258	train-mlogloss:0.998012
[45]	eval-mlogloss:1.01414	train-mlogloss:0.989314
[46]	eval-mlogloss:1.00579	train-mlogloss:0.980936
[47]	eval-mlogloss:0.997731	train-mlogloss:0.972935
[48]	eval-mlogloss:0.989834	train-mlogloss:0.965023
[49]	eval-mlogloss:0.982429	train-mlogloss:0.957686
[50]	eval-mlogloss:0.975079	train-mlogloss:0.950463
[51]	eval-mlogloss:0.968175	train-mlogloss:0.943529
[52]	eval-mlogloss:0.961294	train-mlogloss:0.936699
[53]	eval-mlogloss:0.954523	train-mlogloss:0.929798
[54]	eval-mlogloss:0.947252	train-mlogloss:0.922463
[55]	eval-mlogloss:0.940576	train-mlogloss:0.915937
[56]	eval-mlogloss:0.934156	train-mlogloss:0.909606
[57]	eval-mlogloss:0.92801	train-mlogloss:0.903408
[58]	eval-mlogloss:0.921853	train-mlogloss:0.897206
[59]	eval-mlogloss:0.91565	train-mlogloss:0.890832
[60]	eval-mlogloss:0.909917	train-mlogloss:0.885109
[61]	eval-mlogloss:0.904092	train-mlogloss:0.879381
[62]	eval-mlogloss:0.898742	train-mlogloss:0.873992
[63]	eval-mlogloss:0.893321	train-mlogloss:0.868505
[64]	eval-mlogloss:0.887929	train-mlogloss:0.863103
[65]	eval-mlogloss:0.882554	train-mlogloss:0.857874
[66]	eval-mlogloss:0.877429	train-mlogloss:0.852866
[67]	eval-mlogloss:0.872159	train-mlogloss:0.847537
[68]	eval-mlogloss:0.867255	train-mlogloss:0.842538
[69]	eval-mlogloss:0.862554	train-mlogloss:0.837775
[70]	eval-mlogloss:0.857781	train-mlogloss:0.83289
[71]	eval-mlogloss:0.853161	train-mlogloss:0.828362
[72]	eval-mlogloss:0.848621	train-mlogloss:0.823769
[73]	eval-mlogloss:0.844344	train-mlogloss:0.819451
[74]	eval-mlogloss:0.840193	train-mlogloss:0.815262
[75]	eval-mlogloss:0.835886	train-mlogloss:0.811046
[76]	eval-mlogloss:0.83153	train-mlogloss:0.806908
[77]	eval-mlogloss:0.827361	train-mlogloss:0.802625
[78]	eval-mlogloss:0.823179	train-mlogloss:0.798491
[79]	eval-mlogloss:0.819562	train-mlogloss:0.794775
[80]	eval-mlogloss:0.815588	train-mlogloss:0.79092
[81]	eval-mlogloss:0.811614	train-mlogloss:0.786875
[82]	eval-mlogloss:0.807671	train-mlogloss:0.782808
[83]	eval-mlogloss:0.804046	train-mlogloss:0.779307
[84]	eval-mlogloss:0.799997	train-mlogloss:0.775321
[85]	eval-mlogloss:0.796373	train-mlogloss:0.77182
[86]	eval-mlogloss:0.792859	train-mlogloss:0.76827
[87]	eval-mlogloss:0.788935	train-mlogloss:0.764285
[88]	eval-mlogloss:0.785616	train-mlogloss:0.760846
[89]	eval-mlogloss:0.782346	train-mlogloss:0.757666
[90]	eval-mlogloss:0.77852	train-mlogloss:0.753971
[91]	eval-mlogloss:0.775376	train-mlogloss:0.750848
[92]	eval-mlogloss:0.77243	train-mlogloss:0.747971
[93]	eval-mlogloss:0.768688	train-mlogloss:0.744272
[94]	eval-mlogloss:0.765881	train-mlogloss:0.741433
[95]	eval-mlogloss:0.762257	train-mlogloss:0.737806
[96]	eval-mlogloss:0.759132	train-mlogloss:0.734774
[97]	eval-mlogloss:0.755719	train-mlogloss:0.73128
[98]	eval-mlogloss:0.753011	train-mlogloss:0.72865
[99]	eval-mlogloss:0.750302	train-mlogloss:0.726054
[100]	eval-mlogloss:0.747685	train-mlogloss:0.723404
[101]	eval-mlogloss:0.744771	train-mlogloss:0.720547
[102]	eval-mlogloss:0.741858	train-mlogloss:0.717617
[103]	eval-mlogloss:0.739241	train-mlogloss:0.715021
[104]	eval-mlogloss:0.736166	train-mlogloss:0.711851
[105]	eval-mlogloss:0.733202	train-mlogloss:0.708994
[106]	eval-mlogloss:0.730148	train-mlogloss:0.705729
[107]	eval-mlogloss:0.727643	train-mlogloss:0.703287
[108]	eval-mlogloss:0.725243	train-mlogloss:0.700918
[109]	eval-mlogloss:0.722467	train-mlogloss:0.698127
[110]	eval-mlogloss:0.72018	train-mlogloss:0.695752
[111]	eval-mlogloss:0.716916	train-mlogloss:0.692475
[112]	eval-mlogloss:0.714134	train-mlogloss:0.689736
[113]	eval-mlogloss:0.711753	train-mlogloss:0.687321
[114]	eval-mlogloss:0.709637	train-mlogloss:0.685305
[115]	eval-mlogloss:0.707224	train-mlogloss:0.682917
[116]	eval-mlogloss:0.70454	train-mlogloss:0.680312
[117]	eval-mlogloss:0.702331	train-mlogloss:0.678075
[118]	eval-mlogloss:0.700121	train-mlogloss:0.675841
[119]	eval-mlogloss:0.697599	train-mlogloss:0.673163
[120]	eval-mlogloss:0.695358	train-mlogloss:0.670933
[121]	eval-mlogloss:0.693005	train-mlogloss:0.668471
	Score 0.768206149633


Training with params: 
{'colsample_bytree': 0.75, 'silent': 1, 'eval_metric': 'mlogloss', 'nthread': 4, 'min_child_weight': 5.0, 'n_estimators': 111.0, 'subsample': 0.9500000000000001, 'eta': 0.05, 'objective': 'multi:softmax', 'num_class': 9, 'tree_method': 'exact', 'seed': 314159265, 'max_depth': 11, 'gamma': 0.6000000000000001, 'booster': 'gbtree'}
[0]	eval-mlogloss:2.03446	train-mlogloss:2.02775
[1]	eval-mlogloss:1.90144	train-mlogloss:1.887
[2]	eval-mlogloss:1.78159	train-mlogloss:1.76181
[3]	eval-mlogloss:1.67796	train-mlogloss:1.65274
[4]	eval-mlogloss:1.58725	train-mlogloss:1.55808
[5]	eval-mlogloss:1.50414	train-mlogloss:1.47117
[6]	eval-mlogloss:1.42699	train-mlogloss:1.39104
[7]	eval-mlogloss:1.35763	train-mlogloss:1.31881
[8]	eval-mlogloss:1.29347	train-mlogloss:1.25197
[9]	eval-mlogloss:1.23314	train-mlogloss:1.18961
[10]	eval-mlogloss:1.17794	train-mlogloss:1.13249
[11]	eval-mlogloss:1.12824	train-mlogloss:1.08104
[12]	eval-mlogloss:1.07975	train-mlogloss:1.03109
[13]	eval-mlogloss:1.03509	train-mlogloss:0.985164
[14]	eval-mlogloss:0.991645	train-mlogloss:0.940336
[15]	eval-mlogloss:0.952656	train-mlogloss:0.899988
[16]	eval-mlogloss:0.914948	train-mlogloss:0.861099
[17]	eval-mlogloss:0.878774	train-mlogloss:0.824275
[18]	eval-mlogloss:0.845551	train-mlogloss:0.790457
[19]	eval-mlogloss:0.813629	train-mlogloss:0.758231
[20]	eval-mlogloss:0.783945	train-mlogloss:0.727982
[21]	eval-mlogloss:0.754889	train-mlogloss:0.69845
[22]	eval-mlogloss:0.727716	train-mlogloss:0.670841
[23]	eval-mlogloss:0.701214	train-mlogloss:0.643801
[24]	eval-mlogloss:0.676292	train-mlogloss:0.618775
[25]	eval-mlogloss:0.652963	train-mlogloss:0.595203
[26]	eval-mlogloss:0.630865	train-mlogloss:0.573073
[27]	eval-mlogloss:0.609622	train-mlogloss:0.551612
[28]	eval-mlogloss:0.589202	train-mlogloss:0.531131
[29]	eval-mlogloss:0.569512	train-mlogloss:0.511373
[30]	eval-mlogloss:0.551307	train-mlogloss:0.492872
[31]	eval-mlogloss:0.532995	train-mlogloss:0.474589
[32]	eval-mlogloss:0.515849	train-mlogloss:0.457602
[33]	eval-mlogloss:0.499246	train-mlogloss:0.440991
[34]	eval-mlogloss:0.483824	train-mlogloss:0.425514
[35]	eval-mlogloss:0.469116	train-mlogloss:0.410902
[36]	eval-mlogloss:0.454707	train-mlogloss:0.396713
[37]	eval-mlogloss:0.440984	train-mlogloss:0.383148
[38]	eval-mlogloss:0.427716	train-mlogloss:0.370108
[39]	eval-mlogloss:0.41439	train-mlogloss:0.357031
[40]	eval-mlogloss:0.401768	train-mlogloss:0.344558
[41]	eval-mlogloss:0.390108	train-mlogloss:0.333136
[42]	eval-mlogloss:0.379102	train-mlogloss:0.322379
[43]	eval-mlogloss:0.367946	train-mlogloss:0.311518
[44]	eval-mlogloss:0.357047	train-mlogloss:0.300866
[45]	eval-mlogloss:0.34719	train-mlogloss:0.291151
[46]	eval-mlogloss:0.337781	train-mlogloss:0.282026
[47]	eval-mlogloss:0.328786	train-mlogloss:0.273245
[48]	eval-mlogloss:0.320199	train-mlogloss:0.264949
[49]	eval-mlogloss:0.312235	train-mlogloss:0.257292
[50]	eval-mlogloss:0.303435	train-mlogloss:0.248937
[51]	eval-mlogloss:0.295131	train-mlogloss:0.241009
[52]	eval-mlogloss:0.287726	train-mlogloss:0.233949
[53]	eval-mlogloss:0.28008	train-mlogloss:0.226693
[54]	eval-mlogloss:0.272578	train-mlogloss:0.21952
[55]	eval-mlogloss:0.265978	train-mlogloss:0.213237
[56]	eval-mlogloss:0.25963	train-mlogloss:0.207253
[57]	eval-mlogloss:0.252913	train-mlogloss:0.200903
[58]	eval-mlogloss:0.246327	train-mlogloss:0.194619
[59]	eval-mlogloss:0.239919	train-mlogloss:0.188557
[60]	eval-mlogloss:0.233828	train-mlogloss:0.182778
[61]	eval-mlogloss:0.228124	train-mlogloss:0.177394
[62]	eval-mlogloss:0.222376	train-mlogloss:0.172052
[63]	eval-mlogloss:0.216752	train-mlogloss:0.166842
[64]	eval-mlogloss:0.211886	train-mlogloss:0.162384
[65]	eval-mlogloss:0.207191	train-mlogloss:0.158042
[66]	eval-mlogloss:0.202443	train-mlogloss:0.153675
[67]	eval-mlogloss:0.197674	train-mlogloss:0.149255
[68]	eval-mlogloss:0.193472	train-mlogloss:0.145341
[69]	eval-mlogloss:0.189116	train-mlogloss:0.141286
[70]	eval-mlogloss:0.184565	train-mlogloss:0.137142
[71]	eval-mlogloss:0.180243	train-mlogloss:0.133238
[72]	eval-mlogloss:0.176328	train-mlogloss:0.129676
[73]	eval-mlogloss:0.172676	train-mlogloss:0.126376
[74]	eval-mlogloss:0.169237	train-mlogloss:0.123234
[75]	eval-mlogloss:0.165685	train-mlogloss:0.120074
[76]	eval-mlogloss:0.162179	train-mlogloss:0.116875
[77]	eval-mlogloss:0.159039	train-mlogloss:0.114072
[78]	eval-mlogloss:0.155939	train-mlogloss:0.111254
[79]	eval-mlogloss:0.153069	train-mlogloss:0.108645
[80]	eval-mlogloss:0.15001	train-mlogloss:0.105904
[81]	eval-mlogloss:0.147077	train-mlogloss:0.103294
[82]	eval-mlogloss:0.1441	train-mlogloss:0.100661
[83]	eval-mlogloss:0.141159	train-mlogloss:0.098088
[84]	eval-mlogloss:0.13857	train-mlogloss:0.095862
[85]	eval-mlogloss:0.135791	train-mlogloss:0.093427
[86]	eval-mlogloss:0.13313	train-mlogloss:0.091148
[87]	eval-mlogloss:0.130765	train-mlogloss:0.08904
[88]	eval-mlogloss:0.128433	train-mlogloss:0.087009
[89]	eval-mlogloss:0.126084	train-mlogloss:0.084956
[90]	eval-mlogloss:0.123997	train-mlogloss:0.083098
[91]	eval-mlogloss:0.12195	train-mlogloss:0.081321
[92]	eval-mlogloss:0.119776	train-mlogloss:0.079413
[93]	eval-mlogloss:0.117661	train-mlogloss:0.07763
[94]	eval-mlogloss:0.115623	train-mlogloss:0.075888
[95]	eval-mlogloss:0.113576	train-mlogloss:0.074135
[96]	eval-mlogloss:0.111627	train-mlogloss:0.072477
[97]	eval-mlogloss:0.109713	train-mlogloss:0.070856
[98]	eval-mlogloss:0.107923	train-mlogloss:0.069278
[99]	eval-mlogloss:0.10639	train-mlogloss:0.067941
[100]	eval-mlogloss:0.104673	train-mlogloss:0.066445
[101]	eval-mlogloss:0.103108	train-mlogloss:0.065115
[102]	eval-mlogloss:0.101482	train-mlogloss:0.063762
[103]	eval-mlogloss:0.10001	train-mlogloss:0.062513
[104]	eval-mlogloss:0.098715	train-mlogloss:0.061421
[105]	eval-mlogloss:0.097057	train-mlogloss:0.060052
[106]	eval-mlogloss:0.095579	train-mlogloss:0.05878
[107]	eval-mlogloss:0.094344	train-mlogloss:0.057746
[108]	eval-mlogloss:0.093048	train-mlogloss:0.056619
[109]	eval-mlogloss:0.091837	train-mlogloss:0.055635
[110]	eval-mlogloss:0.090408	train-mlogloss:0.05446
	Score 0.989833603054


The best hyperparameters are:  

{'colsample_bytree': 0.8500000000000001, 'min_child_weight': 4.0, 'n_estimators': 148.0, 'subsample': 0.9, 'eta': 0.15000000000000002, 'max_depth': 8, 'gamma': 0.75}

















In [59]:



    


# Train and test a classifier
def train_and_test(X_tr, y_tr, X_v, well_v):
    
    # Feature normalization
    scaler = preprocessing.RobustScaler(quantile_range=(25.0, 75.0)).fit(X_tr)
    X_tr = scaler.transform(X_tr)
    X_v = scaler.transform(X_v)
    
    # Train classifier
    #clf = make_pipeline(make_union(VotingClassifier([("est", ExtraTreesClassifier(criterion="gini", max_features=1.0, n_estimators=500))]), FunctionTransformer(lambda X: X)), XGBClassifier(learning_rate=0.73, max_depth=10, min_child_weight=10, n_estimators=500, subsample=0.27))
    #clf =  make_pipeline( KNeighborsClassifier(n_neighbors=5, weights="distance") ) 
    #clf = make_pipeline(MaxAbsScaler(),make_union(VotingClassifier([("est", RandomForestClassifier(n_estimators=500))]), FunctionTransformer(lambda X: X)),ExtraTreesClassifier(criterion="entropy", max_features=0.0001, n_estimators=500))
    # * clf = make_pipeline( make_union(VotingClassifier([("est", BernoulliNB(alpha=60.0, binarize=0.26, fit_prior=True))]), FunctionTransformer(lambda X: X)),RandomForestClassifier(n_estimators=500))
    # ** clf = make_pipeline ( XGBClassifier(learning_rate=0.12, max_depth=3, min_child_weight=10, n_estimators=150, seed = 17, colsample_bytree = 0.9) )
    clf = clf = make_pipeline ( XGBClassifier(learning_rate=0.15, max_depth=8, min_child_weight=4, n_estimators=148, seed = SEED, colsample_bytree = 0.85, subsample = 0.9 , gamma = 0.75) )
    clf.fit(X_tr, y_tr)
    
    # Test classifier
    y_v_hat = clf.predict(X_v)
    
    # Clean isolated facies for each well
    for w in np.unique(well_v):
        y_v_hat[well_v==w] = medfilt(y_v_hat[well_v==w], kernel_size=5)
    
    return y_v_hat

Prediction





In [60]:



    


#Load testing data
test_data = pd.read_csv('../validation_data_nofacies.csv')

# Prepare training data
X_tr = X
y_tr = y

# Augment features
X_tr, padded_rows = augment_features(X_tr, well, depth)

# Removed padded rows
X_tr = np.delete(X_tr, padded_rows, axis=0)
y_tr = np.delete(y_tr, padded_rows, axis=0) - 1

# Prepare test data
well_ts = test_data['Well Name'].values
depth_ts = test_data['Depth'].values
X_ts = test_data[feature_names].values

# Augment features
X_ts, padded_rows = augment_features(X_ts, well_ts, depth_ts)

# Predict test labels
y_ts_hat = train_and_test(X_tr, y_tr, X_ts, well_ts)

# Save predicted labels
test_data['Facies'] = y_ts_hat + 1
test_data.to_csv('Prediction_XXX_Final.csv')



    











In [ ]:



    






    











In [ ]:

Content source: seg/2016-ml-contest

Similar notebooks:

notebook.community | gallery | about