notebook.community

Edit and run

DNN-mk-II

Author: Justin Tan

Deep neural network. Do anything from MNIST to signal classification. With bells and whistles. Heavy modified from vanilla implementation.

June: Added cyclical annealing, parameter exponential moving average. Slight performance boost.

Update 26/06: Moved to GPU cluster.

July: Multi-GPU version

To-do: Model ensemble, benchmark, distributed training (*)





In [1]:



    


!nvidia-smi



    


















    






Tue Jul  4 05:35:01 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.48                 Driver Version: 367.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:05:00.0     Off |                    0 |
| N/A   45C    P0    60W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:06:00.0     Off |                    0 |
| N/A   41C    P0    71W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 0000:84:00.0     Off |                    0 |
| N/A   47C    P0    59W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | 0000:85:00.0     Off |                    0 |
| N/A   38C    P0    74W / 149W |      0MiB / 11439MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

















In [1]:



    


import tensorflow as tf
import numpy as np
import pandas as pd
import time, os

class config(object):
    # Set network parameters
    mode = 'kst'
    channel = 'rho0'
    keep_prob = 0.8
    num_epochs = 256
    batch_size = 512
    n_layers = 8
    hidden_layer_nodes = [1024, 1024, 1024, 512, 512, 512, 256, 256]
    learning_rate = 8e-6
    lr_epoch_decay = 0.99
    ema_decay = 0.999
    cycles = 4 # Number of annealing cycles
    n_classes = 2
    n_gpus = 4
    model = 'selu'

class directories(object):
    data = 'data'
    tensorboard = 'tensorboard'
    checkpoints = 'checkpoints'
    
architecture = '{} - {} | Layers: {} | Dropout: {} | Base LR: {} | Epochs: {}'.format(
    config.channel, config.mode, config.n_layers, config.keep_prob, config.learning_rate, config.num_epochs)

class reader():
    # Iterates over data and returns batches
    def __init__(self, df):
        
        self.df = df
        self.batch_size = config.batch_size
        self.steps_per_epoch = len(df) // config.batch_size
        self.epochs = 0
        self.proceed = True
        self.shuffle()

    def shuffle(self):
        self.df = self.df.sample(frac=1).reset_index(drop=True)
        self.df_X = self.df.drop('labels', axis = 1)
        self.df_y = self.df['labels']
        self.pointer = 0


    def next_batch(self, batch_size):
        if self.pointer + 1 >= self.steps_per_epoch:
            inputs = self.df_X.iloc[self.pointer*batch_size:]
            targets = self.df_y.iloc[self.pointer*batch_size:]
            self.epochs += 1
            self.shuffle()
            self.proceed = False
            
        inputs = self.df_X.iloc[self.pointer*batch_size:(self.pointer+1)*batch_size]
        targets = self.df_y.iloc[self.pointer*batch_size:(self.pointer+1)*batch_size]
        self.pointer += 1
                
        return inputs, targets

Functions for graph construction





In [2]:



    


# SELU helper functions
import numbers
from tensorflow.contrib import layers
from tensorflow.python.framework import ops
from tensorflow.python.framework import tensor_shape
from tensorflow.python.framework import tensor_util
from tensorflow.python.ops import math_ops
from tensorflow.python.ops import random_ops
from tensorflow.python.ops import array_ops
from tensorflow.python.layers import utils

""" When using SELUs you have to keep the following in mind:
# (1) scale inputs to zero mean and unit variance
# (2) use SELUs
# (3) initialize weights with stddev sqrt(1/n)
# (4) use SELU dropout
"""
    
# (1) scale inputs to zero mean and unit variance

# (2) use SELUs
def selu(x):
    with ops.name_scope('elu') as scope:
        alpha = 1.6732632423543772848170429916717
        scale = 1.0507009873554804934193349852946
        return scale*tf.where(x>=0.0, x, alpha*tf.nn.elu(x))
    
# (3) initialize weights with stddev sqrt(1/n)
SELU_initializer = layers.variance_scaling_initializer(factor=1.0, mode='FAN_IN')

# (4) use this dropout
def dropout_selu(x, rate, alpha= -1.7580993408473766, fixedPointMean=0.0, fixedPointVar=1.0,
                 noise_shape=None, seed=None, name=None, training=False):
    """Dropout to a value with rescaling."""

    def dropout_selu_impl(x, rate, alpha, noise_shape, seed, name):
        keep_prob = 1.0 - rate
        x = ops.convert_to_tensor(x, name="x")
        if isinstance(keep_prob, numbers.Real) and not 0 < keep_prob <= 1:
            raise ValueError("keep_prob must be a scalar tensor or a float in the "
                                             "range (0, 1], got %g" % keep_prob)
        keep_prob = ops.convert_to_tensor(keep_prob, dtype=x.dtype, name="keep_prob")
        keep_prob.get_shape().assert_is_compatible_with(tensor_shape.scalar())

        alpha = ops.convert_to_tensor(alpha, dtype=x.dtype, name="alpha")
        alpha.get_shape().assert_is_compatible_with(tensor_shape.scalar())

        if tensor_util.constant_value(keep_prob) == 1:
            return x

        noise_shape = noise_shape if noise_shape is not None else array_ops.shape(x)
        random_tensor = keep_prob
        random_tensor += random_ops.random_uniform(noise_shape, seed=seed, dtype=x.dtype)
        binary_tensor = math_ops.floor(random_tensor)
        ret = x * binary_tensor + alpha * (1-binary_tensor)

        a = math_ops.sqrt(fixedPointVar / (keep_prob *((1-keep_prob) * math_ops.pow(alpha-fixedPointMean,2) + fixedPointVar)))

        b = fixedPointMean - a * (keep_prob * fixedPointMean + (1 - keep_prob) * alpha)
        ret = a * ret + b
        ret.set_shape(x.get_shape())
        return ret

    with ops.name_scope(name, "dropout", [x]) as name:
        return utils.smart_cond(training,
                                lambda: dropout_selu_impl(x, rate, alpha, noise_shape, seed, name),
                                lambda: array_ops.identity(x))



    











In [3]:



    


def load_data(file_name, test_size = 0.05):
    from sklearn.model_selection import train_test_split
    df = pd.read_hdf(file_name, 'df')
    df_X_train, df_X_test, df_y_train, df_y_test = train_test_split(df.drop('labels', axis = 1),
                                                                    df['labels'], test_size = test_size, random_state=42)
    return df_X_train, df_X_test, df_y_train, df_y_test

def save_summary(config, delta_t, train_acc, test_acc, test_auc):
    import json
    summary = {
        'Channel': config.channel,
        'Mode': config.mode,
        'Timestamp': time.strftime('%c'),
        'Model': config.model,
        'Layers': config.n_layers,
        'Batch_size': config.batch_size,
        'Dropout': config.keep_prob,
        'Epochs': config.num_epochs,
        'Time': delta_t,
        'Final train acc': train_acc,
        'Final test acc': test_acc,
        'Final test AUC': test_auc
    }
    # Writing JSON data
    if os.path.isfile('vdnn_summary.json'):
        with open('vdnn_summary.json.', 'r+') as f:
            new = json.load(f)
        new.append(summary)
        with open('vdnn_summary.json', 'w') as f:
            json.dump(new, f, indent = 4)
    else:
        with open('vdnn_summary.json', 'w') as f:
             json.dump([summary], f, indent = 4)

def layer_weights(shape, initializer = tf.contrib.layers.xavier_initializer()):
    # Return weight tensor of given shape using Xavier initialization
    W = tf.get_variable("weights", shape = shape, initializer=initializer)
    return W

def layer_biases(shape, init_value = 0.0):
    # Return bias tensor of given shape with small initialized constant value
    b = tf.get_variable("biases", shape = shape, initializer = tf.constant_initializer(init_value))
    return b
                

def hidden_layer_ops(x, shape, name, keep_prob, activation=tf.nn.relu):
    # Add operations to graph to construct hidden layers
    with tf.variable_scope(name) as scope:
        # scope.reuse_variables() # otherwise tf.get_variable() checks that already existing vars are not shared by accident
        weights = layer_weights(shape = shape)
        biases = layer_biases(shape = [shape[1]])
        
        # Apply non-linearity. Default is ReLU
        actv = activation(tf.matmul(x, weights) + biases)
        layer_output = tf.nn.dropout(actv, keep_prob)
        
    return layer_output

def hidden_SELU_ops(x, shape, name, keep_prob, phase = True):
    # Add operations to graph to construct hidden layers
    with tf.variable_scope(name) as scope:
        # scope.reuse_variables() # otherwise tf.get_variable() checks that already existing vars are not shared by accident
        weights = layer_weights(shape = shape, initializer = SELU_initializer)
        biases = layer_biases(shape = [shape[1]])
        
        # Apply non-linearity. Default is ReLU
        actv = selu(tf.add(tf.matmul(x, weights), biases))
        layer_output = dropout_selu(actv, rate = 1 - keep_prob, training = phase)
        
    return layer_output

def BN_layer_ops(x, shape, name, keep_prob, phase, activation=tf.nn.relu):
    # High-level implementation of BN
    with tf.variable_scope(name) as scope:
         # scope.reuse_variables() # otherwise tf.get_variable() checks that already existing vars are not shared by accident
        weights = layer_weights(shape = shape)
        biases = layer_biases(shape = [shape[1]])
        z_BN = tf.matmul(x, weights) + biases
        
        # Place BN transform before non-linearity - update to TF 1.2!
        theta_BN = tf.contrib.layers.batch_norm(z_BN, center=True, scale=True,is_training=phase, 
                                                decay=0.99, zero_debias_moving_mean=True, scope='bn', fused = True)
        BN_actv = activation(theta_BN)
        BN_layer_output = tf.nn.dropout(BN_actv, keep_prob)

    return BN_layer_output

def SELU_BN_layer_ops(x, shape, name, keep_prob, phase):
    # High-level implementation of BN
    with tf.variable_scope(name) as scope:
         # scope.reuse_variables() # otherwise tf.get_variable() checks that already existing vars are not shared by accident
        weights = layer_weights(shape = shape)
        biases = layer_biases(shape = [shape[1]])
        z_BN = tf.matmul(x, weights) + biases
        
        # Place BN transform before non-linearity - update to TF 1.2!
        theta_BN = tf.contrib.layers.batch_norm(z_BN, center=True, scale=True,is_training=phase, 
                                                decay=0.99, zero_debias_moving_mean=True, scope='bn', fused = True)
        BN_actv = selu(theta_BN)
        BN_layer_output = dropout_selu(BN_actv, rate = 1 - keep_prob, training = phase)

    return BN_layer_output

def readout_ops(x, shape, name, initializer = tf.contrib.layers.xavier_initializer()):
    # Don't apply non-linearity, dropout on output layer
    with tf.variable_scope(name) as scope:
        weights = layer_weights(shape = shape, initializer = initializer)
        biases = layer_biases(shape = [shape[1]])
        layer_output = tf.matmul(x, weights) + biases
        
    return layer_output

def build_network(x, n_layers, hidden_layer_nodes, keep_prob, training_phase):
    assert n_layers == len(hidden_layer_nodes), 'Specified layer nodes and number of layers do not correspond.'
    layers = [x]
    with tf.variable_scope('BN_layers') as scope:
        hidden_1 = BN_layer_ops(x, shape = [config.n_features, hidden_layer_nodes[0]], name = 'BNhidden0',
                                keep_prob = keep_prob, phase = training_phase)
        layers.append(hidden_1)
        for n in range(0,n_layers-1):
            hidden_n = BN_layer_ops(layers[-1], shape = [hidden_layer_nodes[n], hidden_layer_nodes[n+1]], name = 'BNhidden{}'.format(n+1),
                                   keep_prob = keep_prob, phase = training_phase)
            layers.append(hidden_n)
        readout = readout_ops(layers[-1], shape = [hidden_layer_nodes[-1], config.n_classes], name = 'readout')
        
    return readout

def network_builder(x, n_layers, hidden_layer_nodes, keep_prob, training_phase):
    assert n_layers == len(hidden_layer_nodes), 'Specified layer nodes and number of layers do not correspond.'
    layers = [x]
    if config.model == 'bn':
        print('Building ReLU + Batch-norm architecture')
        builder = BN_layer_ops
    elif config.model == 'selu':
        print('Building SELU architecture')
        builder = hidden_SELU_ops
    elif config.model == 'selu-bn':
        print('Building SELU + Batch-norm architecture')
        builder = SELU_BN_layer_ops
    else:
        print('Default architecture: SELU')
        builder = hidden_SELU_ops
        
    with tf.variable_scope('hidden_layers') as scope:
        hidden_1 = builder(x, shape = [config.n_features, hidden_layer_nodes[0]], name = 'hidden0',
                                keep_prob = keep_prob, phase = training_phase)
        layers.append(hidden_1)
        for n in range(0,n_layers-1):
            hidden_n = builder(layers[-1], shape = [hidden_layer_nodes[n], hidden_layer_nodes[n+1]], name = 'hidden{}'.format(n+1),
                                   keep_prob = keep_prob, phase = training_phase)
            layers.append(hidden_n)
        readout = readout_ops(layers[-1], shape = [hidden_layer_nodes[-1], config.n_classes], name = 'readout', initializer = SELU_initializer)
        
        return readout

def plot_ROC_curve(network_output, y_true, meta = ''):
#     import matplotlib as mpl
#     mpl.use('pgf')
    import matplotlib.pyplot as plt
    import seaborn as sns
    from sklearn.metrics import roc_curve, auc
    y_score = network_output[:,1]
    
    # Compute ROC curve, integrate
    fpr, tpr, thresholds = roc_curve(y_true, y_score)    
    roc_auc = auc(fpr, tpr)
    
    plt.figure()
    plt.axes([.1,.1,.8,.7])
    plt.figtext(.5,.9, r'$\mathrm{Receiver \;Operating \;Characteristic}$', fontsize=15, ha='center')
    plt.figtext(.5,.85, meta, fontsize=10,ha='center')
    plt.plot(fpr, tpr, color='darkorange',
                     lw=2, label='ROC (area = %0.3f)' % roc_auc)
    plt.plot([0, 1], [0, 1], color='navy', lw=1.0, linestyle='--')
    plt.xlim([-0.05, 1.05])
    plt.ylim([-0.05, 1.05])
    plt.xlabel(r'$\mathrm{False \;Positive \;Rate}$')
    plt.ylabel(r'$\mathrm{True \;Positive \;Rate}$')
    plt.legend(loc="lower right")
    plt.savefig(os.path.join('graphs', '{}_{}_ROC.pdf'.format(config.channel, config.mode)), format='pdf', dpi=1000)
    #plt.savefig(os.path.join('graphs', '{}_{}_ROC.pgf'.format(config.channel, config.mode)), format='pgf', dpi=1000)
    print('AUC: {:g}'.format(roc_auc))
    plt.show()
    plt.gcf().clear()
    return roc_auc
    
def cosine_anneal(initial_lr, t, T, M):
    from math import ceil
    beta = initial_lr/2 * (np.cos(np.pi* (t % ceil(T/M))/ceil(T/M)) + 1)
    return beta

def tower_computation(scope, inputs, labels, keep_prob, training_phase, n_gpu):
    """ Calculate the total loss on a single computation tower.
    Args:
        scope: unique prefix string identifying the tower, e.g. 'tower_0'
        inputs: 2D tensor of shape [batch_size, n_features]
        labels: 1D tensor of shape [batch_size]
    Returns:
        cross_entropy: Tensor containing the total loss for a batch of data
        readout: logits of readout layer
    """
    # Build inference graph
    readout = network_builder(inputs, config.n_layers, config.hidden_layer_nodes, keep_prob, training_phase)
    # Get losses - try L2 loss?
    cross_entropy = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits = readout, labels = labels))
    tf.add_to_collection('losses_collection', cross_entropy)
    
    # Assemble all of the losses for the current tower only.
    losses = tf.get_collection('losses_collection', scope)
    
    # Compute the moving average of all individual losses and the total loss.
#     loss_averages = tf.train.ExponentialMovingAverage(0.9, name='avg')
#     loss_averages_op = loss_averages.apply(losses)

    for l in losses:
        tf.summary.scalar('xentropy_{}-raw'.format(n_gpu), l)
        #tf.summary.scalar('xentropy_{}'.format(n_gpu), loss_averages.average(l))
        
    return cross_entropy, readout

def average_gradients(tower_grads):
    """ Calculate the average gradient for each shared variable across all towers.
    Args:
    tower_grads: Nested list of (gradient, variable) tuples. The outer list
      is over individual gradients. The inner list is over the gradient
      calculation for each tower.
    Returns:
     List of pairs of (gradient, variable) where the gradient has been averaged
     across all towers.
    """
    average_grads = []
    for grad_var_pair in zip(*tower_grads):
        # Note that each grad_and_vars looks like the following:
        #   ((grad0_gpu0, var0_gpu0), ... , (grad0_gpuN, var0_gpuN))
        grads = []
        for g, _ in grad_var_pair:
            # Add 0 dimension to the gradients to represent the tower.
            expanded_g = tf.expand_dims(g, 0)
            # Append on a 'tower' dimension which we will average over below.
            grads.append(expanded_g)
        # Average over the 'tower' dimension.
        grad = tf.concat(axis=0, values=grads)
        grad = tf.reduce_mean(grad, 0)

        # Keep in mind that the Variables are redundant because they are shared
        # across towers. So just return the first tower's pointer to
        # the Variable.
        v = grad_var_pair[0][1]
        gv_pair = (grad, v)
        average_grads.append(gv_pair)
        
    return average_grads



    











In [4]:



    


test_file = '/data/projects/punim0011/jtan/data/dnn/norm_std_dnn_B02rho0gamma_kst.h5'
assert config.batch_size % config.n_gpus == 0, 'Batch size must be divisible by number of GPUs'
assert config.n_layers == len(config.hidden_layer_nodes), 'Inconsistent number of hidden layers'

df_X_train, df_X_test, df_y_train, df_y_test = load_data(test_file)
df_y_train = df_y_train.astype(np.int8)
df_y_test = df_y_test.astype(np.int8)
df_train = pd.concat([df_X_train, df_y_train], axis = 1)
df_test = pd.concat([df_X_test, df_y_test], axis = 1)
config.n_features = df_train.shape[1]-1
config.steps_per_epoch = df_train.shape[0] // config.batch_size
config.T = config.steps_per_epoch*config.num_epochs

readerTrain = reader(df_train)
readerTest = reader(df_test)



    











In [5]:



    


class vanillaDNN():
    # Builds the computational graph
    def __init__(self, config, training = True):
        self.x = tf.placeholder(tf.float32, shape = [None, config.n_features])
        self.y_true = tf.placeholder(tf.int32, shape = None)
        self.keep_prob = tf.placeholder(tf.float32)
        self.training_phase = tf.placeholder(tf.bool)
        self.beta = tf.placeholder(tf.float32)
        self.global_step = tf.get_variable('global_step', [], initializer=tf.constant_initializer(0), trainable=False)
        beta = tf.train.exponential_decay(config.learning_rate, self.global_step, 
                                          decay_steps = config.steps_per_epoch, decay_rate = config.lr_epoch_decay, staircase=True)

        input_batches = tf.split(self.x, config.n_gpus, axis = 0)
        label_batches = tf.split(self.y_true, config.n_gpus, axis = 0)
        opt = tf.train.AdamOptimizer(config.learning_rate)

        # Calculate gradients for each model tower
        tower_grads, tower_readouts, tower_losses, tower_summaries = [], [], [], []
        for gpu in range(config.n_gpus):
            with tf.device('/gpu:{}'.format(gpu)):
                with tf.variable_scope('vDNN', reuse=(gpu > 0)):
                    with tf.name_scope('tower_{}'.format(gpu)) as scope:
                        # Load one batch per GPU
                        input_batch, label_batch = input_batches[gpu], label_batches[gpu]

                        # Calculate loss for one tower of the model. Construct the entire model,
                        # but share the variable across all towers
                        loss, readout = tower_computation(scope, input_batch, label_batch, self.keep_prob, self.training_phase, gpu)
                        # Reuse variables for the next tower, retain the summaries from the final tower.
                        #tf.get_variable_scope().reuse_variables()
                        summaries = tf.get_collection(tf.GraphKeys.SUMMARIES, scope)

                        # Retain batch norm update operations only from the final tower.
                        # batchnorm_updates = tf.get_collection(tf.GraphKeys.UPDATE_OPS, scope)

                        # Calculate the gradients for given batch on this tower
                        grads = opt.compute_gradients(loss)
                        tower_grads.append(grads)
                        tower_readouts.append(readout)
                        tower_summaries.append(summaries)
                        tower_losses.append(loss)

        # Synchronize all towers
        mean_grads = average_gradients(tower_grads)
        self.readout = tf.concat(tower_readouts, axis = 0)

        # Evaluation metrics
        self.cross_entropy = tf.reduce_mean(tower_losses)
        self.prediction = tf.nn.softmax(self.readout)
        correct_prediction = tf.equal(tf.cast(tf.argmax(self.readout, 1), tf.int32), self.y_true)
        self.accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
        _, self.auc_op = tf.metrics.auc(predictions = tf.argmax(self.readout,1), labels = self.y_true, num_thresholds = 512)

        # Track moving average of trainable variables
        self.ema = tf.train.ExponentialMovingAverage(decay = config.ema_decay, num_updates = self.global_step)
        maintain_averages_op = self.ema.apply(tf.trainable_variables())

        # Apply the gradients to adjust the shared variables.
        apply_gradient_op = opt.apply_gradients(mean_grads, global_step=self.global_step)

        # Group all updates to into a single train op.
        #batchnorm_updates_op = tf.group(*batchnorm_updates)
        self.train_op = tf.group(apply_gradient_op, maintain_averages_op)#, batchnorm_updates_op)

        saver = tf.train.Saver(tf.global_variables())

        # Build the summary operation from the last tower summaries
        tower_summaries.append(tf.summary.scalar('cross_entropy', self.cross_entropy))
        tower_summaries.append(tf.summary.scalar('accuracy', self.accuracy))
        tower_summaries.append(tf.summary.scalar('auc', self.auc_op))
        tower_summaries.append(tf.summary.scalar('global_step', self.global_step))
        tower_summaries.append(tf.summary.scalar('learning_rate', beta))
        self.merge_op = tf.summary.merge(tower_summaries)
        self.train_writer = tf.summary.FileWriter(
            os.path.join(directories.tensorboard, 'train_{}'.format(time.strftime('%d-%m_%I:%M'))), graph = tf.get_default_graph())
        self.test_writer = tf.summary.FileWriter(
            os.path.join(directories.tensorboard, 'test_{}'.format(time.strftime('%d-%m_%I:%M'))))
    
    # Make predictions
    def predict(self, ckpt, metaGraph = None):
        pin_cpu = tf.ConfigProto(allow_soft_placement=True, log_device_placement=True, device_count = {'GPU':0})
    
        # Restore the moving average version of the learned variables for eval.
        #variable_averages = tf.train.ExponentialMovingAverage(config.ema_decay)
        variables_to_restore = self.ema.variables_to_restore()
        #variables_to_restore = variable_averages.variables_to_restore()
        saver = tf.train.Saver(variables_to_restore)
        
        with tf.Session(config=pin_cpu) as sess:
            # Initialize variables
            init_op = tf.global_variables_initializer()
            sess.run(init_op)
            sess.run(tf.local_variables_initializer())

            start_time = time.time()
            
            assert (ckpt.model_checkpoint_path or metaGraph), 'Missing checkpoint file!'
            
            if metaGraph:
                saver = tf.train.import_meta_graph(metaGraph)
                saver.restore(sess, os.path.splitext(metaGraph)[0])
                print('{} restored.'.format(metaGraph))
            else:    
                saver.restore(sess, ckpt.model_checkpoint_path)
                print('{} restored.'.format(ckpt.model_checkpoint_path))

            # Make predictions using the trained model
            feed_dict_test = {self.x: df_X_test.values, self.y_true: df_y_test.values, self.keep_prob: 1.0, self.training_phase: False}
            network_output_test, final_v_acc, final_v_auc = sess.run(
                [self.prediction, self.accuracy, self.auc_op], feed_dict = feed_dict_test)

            print("Validation accuracy: {:g}\nValidation AUC: {:g}".format(final_v_acc, final_v_auc))
            
            plot_ROC_curve(network_output = network_output_test, y_true = df_y_test.values,
                           meta = architecture + ' | Test accuracy: {}'.format(final_v_acc))
            delta_t = time.time() - start_time
            print("Inference complete. Duration: %g s" %(delta_t))
            
            return network_output_test



    











In [ ]:



    


def train(config, restore = False):
    # Executes training operations
    vDNN = vanillaDNN(config, training = True)
    start_time = time.time()
    v_acc_best = 0.
    global_step = 0
    global_epoch = 0
    saver = tf.train.Saver()
    ckpt = tf.train.get_checkpoint_state(directories.checkpoints)

    with tf.Session(config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)) as sess:
        # Initialize variables
        init_op = tf.global_variables_initializer()
        sess.run(init_op)
        
        if restore and ckpt.model_checkpoint_path:
            saver.restore(sess, ckpt.model_checkpoint_path)
            global_epoch = ckpt.model_checkpoint_path.split('/')[-1].split('-')[-1]
            assert (type(global_epoch) == int), 'Epoch number untracked'
            print('{} restored at epoch {}.'.format(ckpt.model_checkpoint_path, global_epoch))              
            
        for epoch in range(global_epoch,config.num_epochs):
            
            readerTrain.proceed = True
            step = 0
            # Save every 10 epochs    
            if epoch % 10 == 0:
                save_path = saver.save(sess,
                                       os.path.join(directories.checkpoints,
                                                    'vDNN_{}_{}_epoch{}.ckpt'.format(config.mode, config.channel, epoch)),
                                       global_step = epoch)
                print('Graph saved to file: {}'.format(save_path))
                
            print('(*) Entering Epoch {} ({:.3f} s)'.format(epoch, time.time() - start_time))
            
            while(readerTrain.proceed):
                # Iterate through entire corpus
                x_train, y_train = readerTrain.next_batch(config.batch_size)
                beta = cosine_anneal(config.learning_rate, global_step, config.T, config.cycles)
                feed_dict_train = {vDNN.x: x_train.values, vDNN.y_true: y_train.values, 
                                   vDNN.keep_prob: config.keep_prob, vDNN.training_phase: True, vDNN.beta: beta}
                t_op = sess.run(vDNN.train_op, feed_dict = feed_dict_train)
                step += 1
                global_step += 1

                if step % (config.steps_per_epoch // 5) == 0:            
                    # Evaluate model
                    improved = ''
                    sess.run(tf.local_variables_initializer())
                    x_test, y_test = readerTest.next_batch(config.batch_size)
                    feed_dict_test = {vDNN.x: x_test.values, vDNN.y_true: y_test.values, vDNN.keep_prob: 1.0,
                                      vDNN.training_phase: False, vDNN.beta: 0}

                    t_acc, t_summary = sess.run([vDNN.accuracy, vDNN.merge_op],
                                                        feed_dict = feed_dict_train)
                    v_acc, v_loss, v_auc, v_summary, = sess.run([vDNN.accuracy, vDNN.cross_entropy, vDNN.auc_op, vDNN.merge_op],
                                                        feed_dict = feed_dict_test)

                    vDNN.train_writer.add_summary(t_summary, step)
                    vDNN.test_writer.add_summary(v_summary, step)
                    
                    if epoch > 8 and v_acc > v_acc_best:
                        v_acc_best = v_acc
                        improved = '[*]'
                        save_path = saver.save(sess, 
                                               os.path.join(directories.checkpoints,
                                                            'vDNN_{}_{}_best.ckpt'.format(config.mode, config.channel)),
                                               global_step = epoch)
                    
                    print('Epoch {}, Step {} | Training Acc: {:.3f} | Test Acc: {:.3f} | Test Loss: {:.3f} | Test AUC {:.3f} ({:.2f} s) {}'
                          .format(epoch, step, t_acc, v_acc, v_loss, v_auc, time.time() - start_time, improved))

        save_path = saver.save(sess, os.path.join(directories.checkpoints, 'vDNN_{}_{}_end.ckpt'.format(config.mode, config.channel)),
                               global_step = epoch)
        print('Model saved to file: {}'.format(save_path))

# #         end = int(config.n_gpus*round(df_X_train.shape[0]/config.n_gpus))
# #         feed_dict_train = {vDNN.x: df_X_train.iloc[:end].values, vDNN.y_true: df_y_train.iloc[:end].values, vDNN.keep_prob: 1.0, vDNN.training_phase: False}
# #         feed_dict_test = {vDNN.x: df_X_test.iloc[:end].values, vDNN.y_true: df_y_test.iloc[:end].values, vDNN.keep_prob: 1.0, vDNN.training_phase: False}
# #         final_t_acc = vDNN.accuracy.eval(feed_dict = feed_dict_train)
# #         final_v_acc, final_v_AUC = sess.run([vDNN.accuracy, vDNN.auc_op], feed_dict = feed_dict_test
#     print("Train accuracy: {:g}\nValidation accuracy: {:g}\nValidation AUC: {:g}".format(final_t_acc, final_v_acc, final_v_AUC))
#     save_summary(config, delta_t, final_t_acc, final_v_acc, final_v_AUC)

    delta_t = time.time() - start_time        
    print("Training Complete. Time elapsed: {:.3f} s".format(delta_t))
    print('Architecture: {}'.format(architecture))



    











In [ ]:



    


train(config)



    


















    






Building SELU architecture
Building SELU architecture
Building SELU architecture
Building SELU architecture
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch0.ckpt-0
(*) Entering Epoch 0 (11.952 s)
Epoch 0, Step 190 | Training Acc: 0.533 | Test Acc: 0.666 | Test Loss: 1.269 | Test AUC 0.612 (16.95 s) 
Epoch 0, Step 380 | Training Acc: 0.633 | Test Acc: 0.682 | Test Loss: 2.177 | Test AUC 0.667 (20.67 s) 
Epoch 0, Step 570 | Training Acc: 0.678 | Test Acc: 0.771 | Test Loss: 1.816 | Test AUC 0.730 (24.39 s) 
Epoch 0, Step 760 | Training Acc: 0.697 | Test Acc: 0.773 | Test Loss: 2.182 | Test AUC 0.738 (28.09 s) 
Epoch 0, Step 950 | Training Acc: 0.727 | Test Acc: 0.750 | Test Loss: 2.191 | Test AUC 0.741 (32.75 s) 
(*) Entering Epoch 1 (32.754 s)
Epoch 1, Step 190 | Training Acc: 0.742 | Test Acc: 0.744 | Test Loss: 2.095 | Test AUC 0.747 (36.45 s) 
Epoch 1, Step 380 | Training Acc: 0.699 | Test Acc: 0.764 | Test Loss: 1.925 | Test AUC 0.728 (40.20 s) 
Epoch 1, Step 570 | Training Acc: 0.727 | Test Acc: 0.779 | Test Loss: 1.710 | Test AUC 0.749 (43.91 s) 
Epoch 1, Step 760 | Training Acc: 0.709 | Test Acc: 0.773 | Test Loss: 1.689 | Test AUC 0.739 (47.62 s) 
(*) Entering Epoch 2 (52.081 s)
Epoch 2, Step 190 | Training Acc: 0.738 | Test Acc: 0.789 | Test Loss: 1.411 | Test AUC 0.762 (55.78 s) 
Epoch 2, Step 380 | Training Acc: 0.725 | Test Acc: 0.777 | Test Loss: 1.516 | Test AUC 0.745 (59.50 s) 
Epoch 2, Step 570 | Training Acc: 0.744 | Test Acc: 0.766 | Test Loss: 1.612 | Test AUC 0.752 (63.20 s) 
Epoch 2, Step 760 | Training Acc: 0.771 | Test Acc: 0.787 | Test Loss: 1.520 | Test AUC 0.777 (66.92 s) 
(*) Entering Epoch 3 (71.411 s)
Epoch 3, Step 190 | Training Acc: 0.781 | Test Acc: 0.779 | Test Loss: 1.456 | Test AUC 0.781 (75.10 s) 
Epoch 3, Step 380 | Training Acc: 0.768 | Test Acc: 0.803 | Test Loss: 1.226 | Test AUC 0.785 (78.82 s) 
Epoch 3, Step 570 | Training Acc: 0.785 | Test Acc: 0.787 | Test Loss: 1.434 | Test AUC 0.782 (82.55 s) 
Epoch 3, Step 760 | Training Acc: 0.717 | Test Acc: 0.789 | Test Loss: 1.598 | Test AUC 0.748 (86.26 s) 
(*) Entering Epoch 4 (90.717 s)
Epoch 4, Step 190 | Training Acc: 0.754 | Test Acc: 0.795 | Test Loss: 1.396 | Test AUC 0.775 (94.42 s) 
Epoch 4, Step 380 | Training Acc: 0.762 | Test Acc: 0.818 | Test Loss: 1.127 | Test AUC 0.786 (98.09 s) 
Epoch 4, Step 570 | Training Acc: 0.783 | Test Acc: 0.787 | Test Loss: 1.441 | Test AUC 0.781 (101.79 s) 
Epoch 4, Step 760 | Training Acc: 0.779 | Test Acc: 0.844 | Test Loss: 1.025 | Test AUC 0.813 (105.51 s) 
(*) Entering Epoch 5 (109.932 s)
Epoch 5, Step 190 | Training Acc: 0.770 | Test Acc: 0.793 | Test Loss: 1.331 | Test AUC 0.780 (113.64 s) 
Epoch 5, Step 380 | Training Acc: 0.785 | Test Acc: 0.814 | Test Loss: 1.228 | Test AUC 0.794 (117.35 s) 
Epoch 5, Step 570 | Training Acc: 0.783 | Test Acc: 0.834 | Test Loss: 1.005 | Test AUC 0.805 (121.04 s) 
Epoch 5, Step 760 | Training Acc: 0.746 | Test Acc: 0.795 | Test Loss: 1.439 | Test AUC 0.763 (124.77 s) 
(*) Entering Epoch 6 (129.207 s)
Epoch 6, Step 190 | Training Acc: 0.787 | Test Acc: 0.822 | Test Loss: 1.077 | Test AUC 0.799 (132.88 s) 
Epoch 6, Step 380 | Training Acc: 0.785 | Test Acc: 0.830 | Test Loss: 1.144 | Test AUC 0.799 (136.58 s) 
Epoch 6, Step 570 | Training Acc: 0.771 | Test Acc: 0.822 | Test Loss: 1.219 | Test AUC 0.787 (140.26 s) 
Epoch 6, Step 760 | Training Acc: 0.805 | Test Acc: 0.812 | Test Loss: 1.243 | Test AUC 0.799 (143.96 s) 
(*) Entering Epoch 7 (148.408 s)
Epoch 7, Step 190 | Training Acc: 0.791 | Test Acc: 0.830 | Test Loss: 1.087 | Test AUC 0.800 (152.10 s) 
Epoch 7, Step 380 | Training Acc: 0.791 | Test Acc: 0.770 | Test Loss: 1.546 | Test AUC 0.765 (155.79 s) 
Epoch 7, Step 570 | Training Acc: 0.807 | Test Acc: 0.838 | Test Loss: 1.077 | Test AUC 0.808 (159.46 s) 
Epoch 7, Step 760 | Training Acc: 0.797 | Test Acc: 0.809 | Test Loss: 1.150 | Test AUC 0.793 (163.13 s) 
(*) Entering Epoch 8 (167.574 s)
Epoch 8, Step 190 | Training Acc: 0.783 | Test Acc: 0.824 | Test Loss: 1.189 | Test AUC 0.791 (171.28 s) 
Epoch 8, Step 380 | Training Acc: 0.797 | Test Acc: 0.807 | Test Loss: 1.385 | Test AUC 0.788 (174.96 s) 
Epoch 8, Step 570 | Training Acc: 0.824 | Test Acc: 0.805 | Test Loss: 1.364 | Test AUC 0.802 (178.65 s) 
Epoch 8, Step 760 | Training Acc: 0.805 | Test Acc: 0.809 | Test Loss: 1.305 | Test AUC 0.793 (182.34 s) 
(*) Entering Epoch 9 (186.724 s)
Epoch 9, Step 190 | Training Acc: 0.793 | Test Acc: 0.809 | Test Loss: 1.290 | Test AUC 0.785 (190.95 s) [*]
Epoch 9, Step 380 | Training Acc: 0.807 | Test Acc: 0.861 | Test Loss: 1.069 | Test AUC 0.821 (195.33 s) [*]
Epoch 9, Step 570 | Training Acc: 0.799 | Test Acc: 0.830 | Test Loss: 1.270 | Test AUC 0.797 (199.03 s) 
Epoch 9, Step 760 | Training Acc: 0.797 | Test Acc: 0.832 | Test Loss: 1.204 | Test AUC 0.802 (202.72 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch10.ckpt-10
(*) Entering Epoch 10 (207.809 s)
Epoch 10, Step 190 | Training Acc: 0.807 | Test Acc: 0.811 | Test Loss: 1.365 | Test AUC 0.797 (211.50 s) 
Epoch 10, Step 380 | Training Acc: 0.828 | Test Acc: 0.838 | Test Loss: 1.066 | Test AUC 0.819 (215.20 s) 
Epoch 10, Step 570 | Training Acc: 0.801 | Test Acc: 0.820 | Test Loss: 1.371 | Test AUC 0.801 (218.88 s) 
Epoch 10, Step 760 | Training Acc: 0.836 | Test Acc: 0.824 | Test Loss: 1.268 | Test AUC 0.812 (222.57 s) 
(*) Entering Epoch 11 (227.011 s)
Epoch 11, Step 190 | Training Acc: 0.816 | Test Acc: 0.846 | Test Loss: 1.066 | Test AUC 0.820 (230.70 s) 
Epoch 11, Step 380 | Training Acc: 0.799 | Test Acc: 0.834 | Test Loss: 1.127 | Test AUC 0.800 (234.36 s) 
Epoch 11, Step 570 | Training Acc: 0.834 | Test Acc: 0.852 | Test Loss: 0.997 | Test AUC 0.817 (238.03 s) 
Epoch 11, Step 760 | Training Acc: 0.803 | Test Acc: 0.820 | Test Loss: 1.302 | Test AUC 0.799 (241.72 s) 
(*) Entering Epoch 12 (246.172 s)
Epoch 12, Step 190 | Training Acc: 0.820 | Test Acc: 0.834 | Test Loss: 1.180 | Test AUC 0.813 (249.92 s) 
Epoch 12, Step 380 | Training Acc: 0.822 | Test Acc: 0.822 | Test Loss: 1.244 | Test AUC 0.803 (253.62 s) 
Epoch 12, Step 570 | Training Acc: 0.832 | Test Acc: 0.832 | Test Loss: 1.249 | Test AUC 0.819 (257.31 s) 
Epoch 12, Step 760 | Training Acc: 0.779 | Test Acc: 0.816 | Test Loss: 1.371 | Test AUC 0.786 (261.00 s) 
(*) Entering Epoch 13 (265.414 s)
Epoch 13, Step 190 | Training Acc: 0.828 | Test Acc: 0.838 | Test Loss: 1.138 | Test AUC 0.818 (269.11 s) 
Epoch 13, Step 380 | Training Acc: 0.791 | Test Acc: 0.832 | Test Loss: 1.214 | Test AUC 0.799 (272.79 s) 
Epoch 13, Step 570 | Training Acc: 0.842 | Test Acc: 0.830 | Test Loss: 1.328 | Test AUC 0.819 (276.49 s) 
Epoch 13, Step 760 | Training Acc: 0.842 | Test Acc: 0.844 | Test Loss: 1.103 | Test AUC 0.830 (280.20 s) 
(*) Entering Epoch 14 (284.637 s)
Epoch 14, Step 190 | Training Acc: 0.824 | Test Acc: 0.834 | Test Loss: 1.288 | Test AUC 0.812 (288.33 s) 
Epoch 14, Step 380 | Training Acc: 0.836 | Test Acc: 0.822 | Test Loss: 1.108 | Test AUC 0.815 (292.00 s) 
Epoch 14, Step 570 | Training Acc: 0.803 | Test Acc: 0.859 | Test Loss: 0.916 | Test AUC 0.821 (295.68 s) 
Epoch 14, Step 760 | Training Acc: 0.795 | Test Acc: 0.840 | Test Loss: 1.147 | Test AUC 0.802 (299.35 s) 
(*) Entering Epoch 15 (303.786 s)
Epoch 15, Step 190 | Training Acc: 0.795 | Test Acc: 0.842 | Test Loss: 1.181 | Test AUC 0.807 (307.48 s) 
Epoch 15, Step 380 | Training Acc: 0.809 | Test Acc: 0.826 | Test Loss: 1.153 | Test AUC 0.802 (311.19 s) 
Epoch 15, Step 570 | Training Acc: 0.811 | Test Acc: 0.807 | Test Loss: 1.432 | Test AUC 0.788 (314.87 s) 
Epoch 15, Step 760 | Training Acc: 0.816 | Test Acc: 0.838 | Test Loss: 1.219 | Test AUC 0.811 (318.58 s) 
(*) Entering Epoch 16 (323.021 s)
Epoch 16, Step 190 | Training Acc: 0.838 | Test Acc: 0.873 | Test Loss: 1.022 | Test AUC 0.843 (327.28 s) [*]
Epoch 16, Step 380 | Training Acc: 0.826 | Test Acc: 0.803 | Test Loss: 1.495 | Test AUC 0.802 (330.98 s) 
Epoch 16, Step 570 | Training Acc: 0.832 | Test Acc: 0.818 | Test Loss: 1.316 | Test AUC 0.813 (334.66 s) 
Epoch 16, Step 760 | Training Acc: 0.830 | Test Acc: 0.846 | Test Loss: 1.180 | Test AUC 0.826 (338.34 s) 
(*) Entering Epoch 17 (342.784 s)
Epoch 17, Step 190 | Training Acc: 0.795 | Test Acc: 0.836 | Test Loss: 1.186 | Test AUC 0.803 (346.47 s) 
Epoch 17, Step 380 | Training Acc: 0.805 | Test Acc: 0.836 | Test Loss: 1.337 | Test AUC 0.809 (350.16 s) 
Epoch 17, Step 570 | Training Acc: 0.816 | Test Acc: 0.822 | Test Loss: 1.265 | Test AUC 0.804 (353.85 s) 
Epoch 17, Step 760 | Training Acc: 0.842 | Test Acc: 0.855 | Test Loss: 0.867 | Test AUC 0.835 (357.53 s) 
(*) Entering Epoch 18 (361.964 s)
Epoch 18, Step 190 | Training Acc: 0.816 | Test Acc: 0.816 | Test Loss: 1.334 | Test AUC 0.807 (365.65 s) 
Epoch 18, Step 380 | Training Acc: 0.828 | Test Acc: 0.809 | Test Loss: 1.493 | Test AUC 0.800 (369.33 s) 
Epoch 18, Step 570 | Training Acc: 0.857 | Test Acc: 0.863 | Test Loss: 1.049 | Test AUC 0.849 (373.03 s) 
Epoch 18, Step 760 | Training Acc: 0.795 | Test Acc: 0.822 | Test Loss: 1.543 | Test AUC 0.789 (376.70 s) 
(*) Entering Epoch 19 (381.157 s)
Epoch 19, Step 190 | Training Acc: 0.801 | Test Acc: 0.840 | Test Loss: 1.118 | Test AUC 0.807 (384.88 s) 
Epoch 19, Step 380 | Training Acc: 0.832 | Test Acc: 0.838 | Test Loss: 1.293 | Test AUC 0.822 (388.59 s) 
Epoch 19, Step 570 | Training Acc: 0.818 | Test Acc: 0.840 | Test Loss: 1.432 | Test AUC 0.815 (392.28 s) 
Epoch 19, Step 760 | Training Acc: 0.838 | Test Acc: 0.840 | Test Loss: 1.479 | Test AUC 0.825 (395.95 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch20.ckpt-20
(*) Entering Epoch 20 (400.965 s)
Epoch 20, Step 190 | Training Acc: 0.836 | Test Acc: 0.836 | Test Loss: 1.289 | Test AUC 0.820 (404.66 s) 
Epoch 20, Step 380 | Training Acc: 0.818 | Test Acc: 0.855 | Test Loss: 1.068 | Test AUC 0.825 (408.35 s) 
Epoch 20, Step 570 | Training Acc: 0.850 | Test Acc: 0.826 | Test Loss: 1.365 | Test AUC 0.826 (412.07 s) 
Epoch 20, Step 760 | Training Acc: 0.820 | Test Acc: 0.803 | Test Loss: 1.445 | Test AUC 0.790 (415.76 s) 
(*) Entering Epoch 21 (420.206 s)
Epoch 21, Step 190 | Training Acc: 0.832 | Test Acc: 0.836 | Test Loss: 1.323 | Test AUC 0.824 (423.85 s) 
Epoch 21, Step 380 | Training Acc: 0.830 | Test Acc: 0.850 | Test Loss: 1.190 | Test AUC 0.831 (427.54 s) 
Epoch 21, Step 570 | Training Acc: 0.848 | Test Acc: 0.844 | Test Loss: 1.045 | Test AUC 0.829 (431.23 s) 
Epoch 21, Step 760 | Training Acc: 0.834 | Test Acc: 0.863 | Test Loss: 0.957 | Test AUC 0.835 (434.90 s) 
(*) Entering Epoch 22 (439.310 s)
Epoch 22, Step 190 | Training Acc: 0.832 | Test Acc: 0.850 | Test Loss: 1.143 | Test AUC 0.826 (443.00 s) 
Epoch 22, Step 380 | Training Acc: 0.877 | Test Acc: 0.842 | Test Loss: 1.197 | Test AUC 0.842 (446.66 s) 
Epoch 22, Step 570 | Training Acc: 0.850 | Test Acc: 0.801 | Test Loss: 1.523 | Test AUC 0.811 (450.37 s) 
Epoch 22, Step 760 | Training Acc: 0.816 | Test Acc: 0.855 | Test Loss: 1.192 | Test AUC 0.825 (454.05 s) 
(*) Entering Epoch 23 (458.461 s)
Epoch 23, Step 190 | Training Acc: 0.834 | Test Acc: 0.846 | Test Loss: 1.257 | Test AUC 0.824 (462.14 s) 
Epoch 23, Step 380 | Training Acc: 0.844 | Test Acc: 0.850 | Test Loss: 1.206 | Test AUC 0.831 (465.84 s) 
Epoch 23, Step 570 | Training Acc: 0.803 | Test Acc: 0.832 | Test Loss: 1.093 | Test AUC 0.800 (469.56 s) 
Epoch 23, Step 760 | Training Acc: 0.859 | Test Acc: 0.818 | Test Loss: 1.363 | Test AUC 0.825 (473.26 s) 
(*) Entering Epoch 24 (477.708 s)
Epoch 24, Step 190 | Training Acc: 0.848 | Test Acc: 0.824 | Test Loss: 1.290 | Test AUC 0.818 (481.39 s) 
Epoch 24, Step 380 | Training Acc: 0.840 | Test Acc: 0.844 | Test Loss: 1.148 | Test AUC 0.822 (485.13 s) 
Epoch 24, Step 570 | Training Acc: 0.846 | Test Acc: 0.859 | Test Loss: 0.964 | Test AUC 0.841 (488.81 s) 
Epoch 24, Step 760 | Training Acc: 0.854 | Test Acc: 0.846 | Test Loss: 1.200 | Test AUC 0.836 (492.51 s) 
(*) Entering Epoch 25 (496.933 s)
Epoch 25, Step 190 | Training Acc: 0.854 | Test Acc: 0.871 | Test Loss: 0.832 | Test AUC 0.850 (500.65 s) 
Epoch 25, Step 380 | Training Acc: 0.842 | Test Acc: 0.861 | Test Loss: 1.119 | Test AUC 0.839 (504.34 s) 
Epoch 25, Step 570 | Training Acc: 0.836 | Test Acc: 0.840 | Test Loss: 1.277 | Test AUC 0.822 (508.06 s) 
Epoch 25, Step 760 | Training Acc: 0.834 | Test Acc: 0.830 | Test Loss: 1.215 | Test AUC 0.817 (511.75 s) 
(*) Entering Epoch 26 (516.201 s)
Epoch 26, Step 190 | Training Acc: 0.857 | Test Acc: 0.834 | Test Loss: 1.213 | Test AUC 0.830 (519.88 s) 
Epoch 26, Step 380 | Training Acc: 0.824 | Test Acc: 0.826 | Test Loss: 1.316 | Test AUC 0.818 (523.56 s) 
Epoch 26, Step 570 | Training Acc: 0.816 | Test Acc: 0.842 | Test Loss: 1.087 | Test AUC 0.817 (527.26 s) 
Epoch 26, Step 760 | Training Acc: 0.828 | Test Acc: 0.824 | Test Loss: 1.023 | Test AUC 0.813 (530.98 s) 
(*) Entering Epoch 27 (535.418 s)
Epoch 27, Step 190 | Training Acc: 0.883 | Test Acc: 0.830 | Test Loss: 1.212 | Test AUC 0.846 (539.12 s) 
Epoch 27, Step 380 | Training Acc: 0.830 | Test Acc: 0.809 | Test Loss: 1.278 | Test AUC 0.811 (542.81 s) 
Epoch 27, Step 570 | Training Acc: 0.859 | Test Acc: 0.812 | Test Loss: 1.512 | Test AUC 0.827 (546.56 s) 
Epoch 27, Step 760 | Training Acc: 0.811 | Test Acc: 0.854 | Test Loss: 0.963 | Test AUC 0.821 (550.28 s) 
(*) Entering Epoch 28 (554.728 s)
Epoch 28, Step 190 | Training Acc: 0.830 | Test Acc: 0.842 | Test Loss: 1.149 | Test AUC 0.823 (558.44 s) 
Epoch 28, Step 380 | Training Acc: 0.846 | Test Acc: 0.873 | Test Loss: 0.976 | Test AUC 0.855 (562.13 s) 
Epoch 28, Step 570 | Training Acc: 0.844 | Test Acc: 0.838 | Test Loss: 0.978 | Test AUC 0.836 (565.83 s) 
Epoch 28, Step 760 | Training Acc: 0.850 | Test Acc: 0.842 | Test Loss: 0.959 | Test AUC 0.833 (569.54 s) 
(*) Entering Epoch 29 (574.001 s)
Epoch 29, Step 190 | Training Acc: 0.822 | Test Acc: 0.854 | Test Loss: 0.904 | Test AUC 0.825 (577.69 s) 
Epoch 29, Step 380 | Training Acc: 0.846 | Test Acc: 0.840 | Test Loss: 0.924 | Test AUC 0.834 (581.37 s) 
Epoch 29, Step 570 | Training Acc: 0.832 | Test Acc: 0.852 | Test Loss: 0.946 | Test AUC 0.835 (585.04 s) 
Epoch 29, Step 760 | Training Acc: 0.844 | Test Acc: 0.826 | Test Loss: 1.120 | Test AUC 0.828 (588.73 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch30.ckpt-30
(*) Entering Epoch 30 (593.765 s)
Epoch 30, Step 190 | Training Acc: 0.848 | Test Acc: 0.848 | Test Loss: 0.934 | Test AUC 0.840 (597.49 s) 
Epoch 30, Step 380 | Training Acc: 0.828 | Test Acc: 0.857 | Test Loss: 1.117 | Test AUC 0.837 (601.17 s) 
Epoch 30, Step 570 | Training Acc: 0.840 | Test Acc: 0.836 | Test Loss: 0.981 | Test AUC 0.824 (604.86 s) 
Epoch 30, Step 760 | Training Acc: 0.814 | Test Acc: 0.814 | Test Loss: 1.173 | Test AUC 0.803 (608.57 s) 
(*) Entering Epoch 31 (613.018 s)
Epoch 31, Step 190 | Training Acc: 0.848 | Test Acc: 0.820 | Test Loss: 1.104 | Test AUC 0.824 (616.75 s) 
Epoch 31, Step 380 | Training Acc: 0.863 | Test Acc: 0.809 | Test Loss: 1.380 | Test AUC 0.831 (620.46 s) 
Epoch 31, Step 570 | Training Acc: 0.844 | Test Acc: 0.834 | Test Loss: 1.136 | Test AUC 0.831 (624.13 s) 
Epoch 31, Step 760 | Training Acc: 0.830 | Test Acc: 0.838 | Test Loss: 1.130 | Test AUC 0.822 (627.81 s) 
(*) Entering Epoch 32 (632.239 s)
Epoch 32, Step 190 | Training Acc: 0.840 | Test Acc: 0.830 | Test Loss: 0.821 | Test AUC 0.826 (635.93 s) 
Epoch 32, Step 380 | Training Acc: 0.852 | Test Acc: 0.873 | Test Loss: 0.778 | Test AUC 0.854 (639.61 s) 
Epoch 32, Step 570 | Training Acc: 0.852 | Test Acc: 0.820 | Test Loss: 1.114 | Test AUC 0.829 (643.31 s) 
Epoch 32, Step 760 | Training Acc: 0.848 | Test Acc: 0.859 | Test Loss: 0.880 | Test AUC 0.850 (647.03 s) 
(*) Entering Epoch 33 (651.439 s)
Epoch 33, Step 190 | Training Acc: 0.836 | Test Acc: 0.809 | Test Loss: 1.166 | Test AUC 0.817 (655.12 s) 
Epoch 33, Step 380 | Training Acc: 0.844 | Test Acc: 0.816 | Test Loss: 1.314 | Test AUC 0.823 (658.80 s) 
Epoch 33, Step 570 | Training Acc: 0.820 | Test Acc: 0.844 | Test Loss: 1.017 | Test AUC 0.824 (662.52 s) 
Epoch 33, Step 760 | Training Acc: 0.834 | Test Acc: 0.828 | Test Loss: 1.092 | Test AUC 0.822 (666.20 s) 
(*) Entering Epoch 34 (670.679 s)
Epoch 34, Step 190 | Training Acc: 0.848 | Test Acc: 0.865 | Test Loss: 0.910 | Test AUC 0.846 (674.37 s) 
Epoch 34, Step 380 | Training Acc: 0.840 | Test Acc: 0.822 | Test Loss: 1.211 | Test AUC 0.821 (678.04 s) 
Epoch 34, Step 570 | Training Acc: 0.830 | Test Acc: 0.809 | Test Loss: 1.189 | Test AUC 0.806 (681.72 s) 
Epoch 34, Step 760 | Training Acc: 0.844 | Test Acc: 0.824 | Test Loss: 1.110 | Test AUC 0.831 (685.41 s) 
(*) Entering Epoch 35 (689.836 s)
Epoch 35, Step 190 | Training Acc: 0.828 | Test Acc: 0.820 | Test Loss: 1.154 | Test AUC 0.818 (693.57 s) 
Epoch 35, Step 380 | Training Acc: 0.852 | Test Acc: 0.838 | Test Loss: 0.814 | Test AUC 0.842 (697.25 s) 
Epoch 35, Step 570 | Training Acc: 0.850 | Test Acc: 0.834 | Test Loss: 1.067 | Test AUC 0.842 (700.95 s) 
Epoch 35, Step 760 | Training Acc: 0.838 | Test Acc: 0.811 | Test Loss: 1.030 | Test AUC 0.819 (704.60 s) 
(*) Entering Epoch 36 (709.032 s)
Epoch 36, Step 190 | Training Acc: 0.842 | Test Acc: 0.854 | Test Loss: 0.982 | Test AUC 0.842 (712.75 s) 
Epoch 36, Step 380 | Training Acc: 0.869 | Test Acc: 0.871 | Test Loss: 0.841 | Test AUC 0.863 (716.45 s) 
Epoch 36, Step 570 | Training Acc: 0.865 | Test Acc: 0.824 | Test Loss: 1.000 | Test AUC 0.840 (720.21 s) 
Epoch 36, Step 760 | Training Acc: 0.861 | Test Acc: 0.855 | Test Loss: 1.163 | Test AUC 0.857 (723.91 s) 
(*) Entering Epoch 37 (728.363 s)
Epoch 37, Step 190 | Training Acc: 0.855 | Test Acc: 0.838 | Test Loss: 0.965 | Test AUC 0.840 (732.06 s) 
Epoch 37, Step 380 | Training Acc: 0.846 | Test Acc: 0.824 | Test Loss: 1.076 | Test AUC 0.826 (735.76 s) 
Epoch 37, Step 570 | Training Acc: 0.867 | Test Acc: 0.842 | Test Loss: 1.000 | Test AUC 0.849 (739.45 s) 
Epoch 37, Step 760 | Training Acc: 0.861 | Test Acc: 0.840 | Test Loss: 0.897 | Test AUC 0.848 (743.14 s) 
(*) Entering Epoch 38 (747.579 s)
Epoch 38, Step 190 | Training Acc: 0.820 | Test Acc: 0.826 | Test Loss: 1.138 | Test AUC 0.816 (751.29 s) 
Epoch 38, Step 380 | Training Acc: 0.852 | Test Acc: 0.826 | Test Loss: 1.197 | Test AUC 0.837 (754.96 s) 
Epoch 38, Step 570 | Training Acc: 0.857 | Test Acc: 0.824 | Test Loss: 1.217 | Test AUC 0.834 (758.66 s) 
Epoch 38, Step 760 | Training Acc: 0.873 | Test Acc: 0.867 | Test Loss: 0.771 | Test AUC 0.864 (762.36 s) 
(*) Entering Epoch 39 (766.789 s)
Epoch 39, Step 190 | Training Acc: 0.859 | Test Acc: 0.844 | Test Loss: 0.954 | Test AUC 0.845 (770.49 s) 
Epoch 39, Step 380 | Training Acc: 0.838 | Test Acc: 0.842 | Test Loss: 0.937 | Test AUC 0.837 (774.20 s) 
Epoch 39, Step 570 | Training Acc: 0.859 | Test Acc: 0.811 | Test Loss: 1.233 | Test AUC 0.831 (777.88 s) 
Epoch 39, Step 760 | Training Acc: 0.873 | Test Acc: 0.854 | Test Loss: 0.976 | Test AUC 0.859 (781.53 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch40.ckpt-40
(*) Entering Epoch 40 (786.588 s)
Epoch 40, Step 190 | Training Acc: 0.863 | Test Acc: 0.859 | Test Loss: 0.958 | Test AUC 0.857 (790.28 s) 
Epoch 40, Step 380 | Training Acc: 0.844 | Test Acc: 0.844 | Test Loss: 0.988 | Test AUC 0.840 (793.97 s) 
Epoch 40, Step 570 | Training Acc: 0.848 | Test Acc: 0.842 | Test Loss: 1.179 | Test AUC 0.834 (797.66 s) 
Epoch 40, Step 760 | Training Acc: 0.844 | Test Acc: 0.830 | Test Loss: 1.168 | Test AUC 0.831 (801.34 s) 
(*) Entering Epoch 41 (805.770 s)
Epoch 41, Step 190 | Training Acc: 0.850 | Test Acc: 0.824 | Test Loss: 1.083 | Test AUC 0.835 (809.45 s) 
Epoch 41, Step 380 | Training Acc: 0.840 | Test Acc: 0.861 | Test Loss: 0.834 | Test AUC 0.847 (813.12 s) 
Epoch 41, Step 570 | Training Acc: 0.869 | Test Acc: 0.844 | Test Loss: 1.026 | Test AUC 0.853 (816.80 s) 
Epoch 41, Step 760 | Training Acc: 0.840 | Test Acc: 0.855 | Test Loss: 0.781 | Test AUC 0.845 (820.51 s) 
(*) Entering Epoch 42 (824.919 s)
Epoch 42, Step 190 | Training Acc: 0.838 | Test Acc: 0.832 | Test Loss: 1.077 | Test AUC 0.834 (828.58 s) 
Epoch 42, Step 380 | Training Acc: 0.881 | Test Acc: 0.846 | Test Loss: 0.951 | Test AUC 0.861 (832.28 s) 
Epoch 42, Step 570 | Training Acc: 0.854 | Test Acc: 0.885 | Test Loss: 0.766 | Test AUC 0.859 (836.52 s) [*]
Epoch 42, Step 760 | Training Acc: 0.861 | Test Acc: 0.857 | Test Loss: 0.868 | Test AUC 0.855 (840.23 s) 
(*) Entering Epoch 43 (844.687 s)
Epoch 43, Step 190 | Training Acc: 0.869 | Test Acc: 0.830 | Test Loss: 1.043 | Test AUC 0.848 (848.36 s) 
Epoch 43, Step 380 | Training Acc: 0.846 | Test Acc: 0.822 | Test Loss: 1.202 | Test AUC 0.825 (852.04 s) 
Epoch 43, Step 570 | Training Acc: 0.857 | Test Acc: 0.889 | Test Loss: 0.604 | Test AUC 0.873 (856.33 s) [*]
Epoch 43, Step 760 | Training Acc: 0.852 | Test Acc: 0.828 | Test Loss: 1.087 | Test AUC 0.835 (860.01 s) 
(*) Entering Epoch 44 (864.442 s)
Epoch 44, Step 190 | Training Acc: 0.857 | Test Acc: 0.814 | Test Loss: 1.068 | Test AUC 0.829 (868.11 s) 
Epoch 44, Step 380 | Training Acc: 0.852 | Test Acc: 0.826 | Test Loss: 1.285 | Test AUC 0.835 (871.78 s) 
Epoch 44, Step 570 | Training Acc: 0.863 | Test Acc: 0.818 | Test Loss: 1.176 | Test AUC 0.837 (875.47 s) 
Epoch 44, Step 760 | Training Acc: 0.859 | Test Acc: 0.832 | Test Loss: 1.117 | Test AUC 0.839 (879.16 s) 
(*) Entering Epoch 45 (883.595 s)
Epoch 45, Step 190 | Training Acc: 0.830 | Test Acc: 0.818 | Test Loss: 1.286 | Test AUC 0.821 (887.29 s) 
Epoch 45, Step 380 | Training Acc: 0.875 | Test Acc: 0.824 | Test Loss: 1.061 | Test AUC 0.843 (890.96 s) 
Epoch 45, Step 570 | Training Acc: 0.852 | Test Acc: 0.834 | Test Loss: 1.090 | Test AUC 0.837 (894.66 s) 
Epoch 45, Step 760 | Training Acc: 0.883 | Test Acc: 0.854 | Test Loss: 0.840 | Test AUC 0.861 (898.36 s) 
(*) Entering Epoch 46 (902.788 s)
Epoch 46, Step 190 | Training Acc: 0.857 | Test Acc: 0.820 | Test Loss: 1.073 | Test AUC 0.835 (906.45 s) 
Epoch 46, Step 380 | Training Acc: 0.857 | Test Acc: 0.834 | Test Loss: 0.878 | Test AUC 0.847 (910.15 s) 
Epoch 46, Step 570 | Training Acc: 0.850 | Test Acc: 0.832 | Test Loss: 0.915 | Test AUC 0.842 (913.82 s) 
Epoch 46, Step 760 | Training Acc: 0.863 | Test Acc: 0.857 | Test Loss: 0.875 | Test AUC 0.855 (917.53 s) 
(*) Entering Epoch 47 (921.952 s)
Epoch 47, Step 190 | Training Acc: 0.861 | Test Acc: 0.820 | Test Loss: 1.182 | Test AUC 0.838 (925.64 s) 
Epoch 47, Step 380 | Training Acc: 0.850 | Test Acc: 0.828 | Test Loss: 0.932 | Test AUC 0.833 (929.33 s) 
Epoch 47, Step 570 | Training Acc: 0.838 | Test Acc: 0.828 | Test Loss: 0.983 | Test AUC 0.830 (933.03 s) 
Epoch 47, Step 760 | Training Acc: 0.846 | Test Acc: 0.832 | Test Loss: 0.891 | Test AUC 0.834 (936.73 s) 
(*) Entering Epoch 48 (941.166 s)
Epoch 48, Step 190 | Training Acc: 0.871 | Test Acc: 0.844 | Test Loss: 0.902 | Test AUC 0.854 (944.86 s) 
Epoch 48, Step 380 | Training Acc: 0.859 | Test Acc: 0.865 | Test Loss: 0.743 | Test AUC 0.862 (948.54 s) 
Epoch 48, Step 570 | Training Acc: 0.869 | Test Acc: 0.830 | Test Loss: 1.152 | Test AUC 0.845 (952.22 s) 
Epoch 48, Step 760 | Training Acc: 0.832 | Test Acc: 0.838 | Test Loss: 1.052 | Test AUC 0.833 (955.96 s) 
(*) Entering Epoch 49 (960.391 s)
Epoch 49, Step 190 | Training Acc: 0.855 | Test Acc: 0.867 | Test Loss: 0.851 | Test AUC 0.854 (964.07 s) 
Epoch 49, Step 380 | Training Acc: 0.830 | Test Acc: 0.863 | Test Loss: 0.923 | Test AUC 0.842 (967.76 s) 
Epoch 49, Step 570 | Training Acc: 0.850 | Test Acc: 0.838 | Test Loss: 0.990 | Test AUC 0.842 (971.46 s) 
Epoch 49, Step 760 | Training Acc: 0.846 | Test Acc: 0.826 | Test Loss: 1.090 | Test AUC 0.834 (975.15 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch50.ckpt-50
(*) Entering Epoch 50 (980.167 s)
Epoch 50, Step 190 | Training Acc: 0.855 | Test Acc: 0.865 | Test Loss: 0.734 | Test AUC 0.858 (983.82 s) 
Epoch 50, Step 380 | Training Acc: 0.834 | Test Acc: 0.869 | Test Loss: 1.060 | Test AUC 0.848 (987.51 s) 
Epoch 50, Step 570 | Training Acc: 0.867 | Test Acc: 0.842 | Test Loss: 1.018 | Test AUC 0.852 (991.21 s) 
Epoch 50, Step 760 | Training Acc: 0.885 | Test Acc: 0.844 | Test Loss: 1.068 | Test AUC 0.862 (994.88 s) 
(*) Entering Epoch 51 (999.346 s)
Epoch 51, Step 190 | Training Acc: 0.869 | Test Acc: 0.791 | Test Loss: 1.382 | Test AUC 0.829 (1003.04 s) 
Epoch 51, Step 380 | Training Acc: 0.857 | Test Acc: 0.814 | Test Loss: 1.221 | Test AUC 0.832 (1006.74 s) 
Epoch 51, Step 570 | Training Acc: 0.879 | Test Acc: 0.844 | Test Loss: 0.828 | Test AUC 0.860 (1010.46 s) 
Epoch 51, Step 760 | Training Acc: 0.871 | Test Acc: 0.805 | Test Loss: 1.202 | Test AUC 0.834 (1014.12 s) 
(*) Entering Epoch 52 (1018.586 s)
Epoch 52, Step 190 | Training Acc: 0.861 | Test Acc: 0.867 | Test Loss: 0.752 | Test AUC 0.863 (1022.30 s) 
Epoch 52, Step 380 | Training Acc: 0.857 | Test Acc: 0.832 | Test Loss: 1.104 | Test AUC 0.843 (1026.02 s) 
Epoch 52, Step 570 | Training Acc: 0.846 | Test Acc: 0.822 | Test Loss: 1.056 | Test AUC 0.831 (1029.74 s) 
Epoch 52, Step 760 | Training Acc: 0.836 | Test Acc: 0.830 | Test Loss: 1.023 | Test AUC 0.831 (1033.42 s) 
(*) Entering Epoch 53 (1037.880 s)
Epoch 53, Step 190 | Training Acc: 0.857 | Test Acc: 0.791 | Test Loss: 1.128 | Test AUC 0.822 (1041.56 s) 
Epoch 53, Step 380 | Training Acc: 0.854 | Test Acc: 0.836 | Test Loss: 1.058 | Test AUC 0.840 (1045.26 s) 
Epoch 53, Step 570 | Training Acc: 0.871 | Test Acc: 0.844 | Test Loss: 0.913 | Test AUC 0.855 (1048.95 s) 
Epoch 53, Step 760 | Training Acc: 0.854 | Test Acc: 0.822 | Test Loss: 1.223 | Test AUC 0.834 (1052.60 s) 
(*) Entering Epoch 54 (1057.039 s)
Epoch 54, Step 190 | Training Acc: 0.869 | Test Acc: 0.852 | Test Loss: 0.776 | Test AUC 0.861 (1060.76 s) 
Epoch 54, Step 380 | Training Acc: 0.855 | Test Acc: 0.844 | Test Loss: 0.962 | Test AUC 0.848 (1064.40 s) 
Epoch 54, Step 570 | Training Acc: 0.885 | Test Acc: 0.824 | Test Loss: 1.047 | Test AUC 0.852 (1068.09 s) 
Epoch 54, Step 760 | Training Acc: 0.855 | Test Acc: 0.822 | Test Loss: 1.113 | Test AUC 0.835 (1071.78 s) 
(*) Entering Epoch 55 (1076.240 s)
Epoch 55, Step 190 | Training Acc: 0.832 | Test Acc: 0.826 | Test Loss: 0.942 | Test AUC 0.827 (1079.91 s) 
Epoch 55, Step 380 | Training Acc: 0.879 | Test Acc: 0.838 | Test Loss: 1.012 | Test AUC 0.856 (1083.63 s) 
Epoch 55, Step 570 | Training Acc: 0.871 | Test Acc: 0.822 | Test Loss: 1.091 | Test AUC 0.847 (1087.33 s) 
Epoch 55, Step 760 | Training Acc: 0.848 | Test Acc: 0.848 | Test Loss: 0.821 | Test AUC 0.843 (1091.03 s) 
(*) Entering Epoch 56 (1095.486 s)
Epoch 56, Step 190 | Training Acc: 0.867 | Test Acc: 0.834 | Test Loss: 1.043 | Test AUC 0.850 (1099.16 s) 
Epoch 56, Step 380 | Training Acc: 0.852 | Test Acc: 0.834 | Test Loss: 0.926 | Test AUC 0.840 (1102.85 s) 
Epoch 56, Step 570 | Training Acc: 0.869 | Test Acc: 0.844 | Test Loss: 0.967 | Test AUC 0.854 (1106.54 s) 
Epoch 56, Step 760 | Training Acc: 0.879 | Test Acc: 0.855 | Test Loss: 0.883 | Test AUC 0.863 (1110.27 s) 
(*) Entering Epoch 57 (1114.731 s)
Epoch 57, Step 190 | Training Acc: 0.840 | Test Acc: 0.826 | Test Loss: 1.022 | Test AUC 0.831 (1118.42 s) 
Epoch 57, Step 380 | Training Acc: 0.863 | Test Acc: 0.854 | Test Loss: 0.849 | Test AUC 0.856 (1122.12 s) 
Epoch 57, Step 570 | Training Acc: 0.893 | Test Acc: 0.842 | Test Loss: 0.785 | Test AUC 0.864 (1125.80 s) 
Epoch 57, Step 760 | Training Acc: 0.857 | Test Acc: 0.834 | Test Loss: 1.084 | Test AUC 0.843 (1129.51 s) 
(*) Entering Epoch 58 (1133.942 s)
Epoch 58, Step 190 | Training Acc: 0.842 | Test Acc: 0.836 | Test Loss: 1.145 | Test AUC 0.837 (1137.61 s) 
Epoch 58, Step 380 | Training Acc: 0.871 | Test Acc: 0.852 | Test Loss: 0.966 | Test AUC 0.862 (1141.33 s) 
Epoch 58, Step 570 | Training Acc: 0.861 | Test Acc: 0.832 | Test Loss: 1.063 | Test AUC 0.846 (1145.04 s) 
Epoch 58, Step 760 | Training Acc: 0.865 | Test Acc: 0.838 | Test Loss: 1.021 | Test AUC 0.847 (1148.75 s) 
(*) Entering Epoch 59 (1153.185 s)
Epoch 59, Step 190 | Training Acc: 0.875 | Test Acc: 0.842 | Test Loss: 1.209 | Test AUC 0.858 (1156.84 s) 
Epoch 59, Step 380 | Training Acc: 0.828 | Test Acc: 0.850 | Test Loss: 0.976 | Test AUC 0.832 (1160.58 s) 
Epoch 59, Step 570 | Training Acc: 0.863 | Test Acc: 0.809 | Test Loss: 1.352 | Test AUC 0.831 (1164.24 s) 

















In [6]:



    


train(config)



    


















    






Graph saved to file: checkpoints/vDNN_kst_rho0_epoch0.ckpt-0
(*) Entering Epoch 0 (13.018 s)
Epoch 0, Step 190 | Training Acc: 0.830 | Test Acc: 0.676 | Test Loss: 0.965 | Test AUC 0.771 (22.18 s) 
Epoch 0, Step 380 | Training Acc: 0.812 | Test Acc: 0.816 | Test Loss: 0.483 | Test AUC 0.815 (29.07 s) 
Epoch 0, Step 570 | Training Acc: 0.818 | Test Acc: 0.814 | Test Loss: 0.453 | Test AUC 0.818 (35.93 s) 
Epoch 0, Step 760 | Training Acc: 0.838 | Test Acc: 0.828 | Test Loss: 0.376 | Test AUC 0.828 (42.77 s) 
Epoch 0, Step 950 | Training Acc: 0.873 | Test Acc: 0.865 | Test Loss: 0.325 | Test AUC 0.868 (50.58 s) 
(*) Entering Epoch 1 (50.583 s)
Epoch 1, Step 190 | Training Acc: 0.850 | Test Acc: 0.865 | Test Loss: 0.341 | Test AUC 0.849 (57.42 s) 
Epoch 1, Step 380 | Training Acc: 0.863 | Test Acc: 0.830 | Test Loss: 0.374 | Test AUC 0.834 (64.25 s) 
Epoch 1, Step 570 | Training Acc: 0.861 | Test Acc: 0.883 | Test Loss: 0.325 | Test AUC 0.863 (71.08 s) 
Epoch 1, Step 760 | Training Acc: 0.879 | Test Acc: 0.861 | Test Loss: 0.301 | Test AUC 0.864 (77.92 s) 
(*) Entering Epoch 2 (85.322 s)
Epoch 2, Step 190 | Training Acc: 0.893 | Test Acc: 0.852 | Test Loss: 0.314 | Test AUC 0.863 (92.17 s) 
Epoch 2, Step 380 | Training Acc: 0.871 | Test Acc: 0.875 | Test Loss: 0.285 | Test AUC 0.864 (99.02 s) 
Epoch 2, Step 570 | Training Acc: 0.850 | Test Acc: 0.877 | Test Loss: 0.294 | Test AUC 0.851 (105.84 s) 
Epoch 2, Step 760 | Training Acc: 0.875 | Test Acc: 0.877 | Test Loss: 0.317 | Test AUC 0.863 (112.64 s) 
(*) Entering Epoch 3 (120.008 s)
Epoch 3, Step 190 | Training Acc: 0.859 | Test Acc: 0.883 | Test Loss: 0.274 | Test AUC 0.858 (126.83 s) 
Epoch 3, Step 380 | Training Acc: 0.861 | Test Acc: 0.879 | Test Loss: 0.282 | Test AUC 0.856 (133.61 s) 
Epoch 3, Step 570 | Training Acc: 0.896 | Test Acc: 0.885 | Test Loss: 0.264 | Test AUC 0.888 (140.37 s) 
Epoch 3, Step 760 | Training Acc: 0.887 | Test Acc: 0.896 | Test Loss: 0.258 | Test AUC 0.883 (147.19 s) 
(*) Entering Epoch 4 (154.531 s)
Epoch 4, Step 190 | Training Acc: 0.871 | Test Acc: 0.885 | Test Loss: 0.299 | Test AUC 0.873 (161.30 s) 
Epoch 4, Step 380 | Training Acc: 0.859 | Test Acc: 0.881 | Test Loss: 0.272 | Test AUC 0.865 (168.09 s) 
Epoch 4, Step 570 | Training Acc: 0.855 | Test Acc: 0.824 | Test Loss: 0.385 | Test AUC 0.814 (174.89 s) 
Epoch 4, Step 760 | Training Acc: 0.855 | Test Acc: 0.848 | Test Loss: 0.349 | Test AUC 0.837 (181.68 s) 
(*) Entering Epoch 5 (189.019 s)
Epoch 5, Step 190 | Training Acc: 0.895 | Test Acc: 0.887 | Test Loss: 0.297 | Test AUC 0.884 (195.81 s) 
Epoch 5, Step 380 | Training Acc: 0.881 | Test Acc: 0.896 | Test Loss: 0.235 | Test AUC 0.886 (202.58 s) 
Epoch 5, Step 570 | Training Acc: 0.887 | Test Acc: 0.898 | Test Loss: 0.245 | Test AUC 0.880 (209.38 s) 
Epoch 5, Step 760 | Training Acc: 0.891 | Test Acc: 0.881 | Test Loss: 0.294 | Test AUC 0.879 (216.17 s) 
(*) Entering Epoch 6 (223.532 s)
Epoch 6, Step 190 | Training Acc: 0.859 | Test Acc: 0.793 | Test Loss: 0.419 | Test AUC 0.797 (230.31 s) 
Epoch 6, Step 380 | Training Acc: 0.869 | Test Acc: 0.842 | Test Loss: 0.366 | Test AUC 0.831 (237.12 s) 
Epoch 6, Step 570 | Training Acc: 0.859 | Test Acc: 0.871 | Test Loss: 0.321 | Test AUC 0.852 (243.90 s) 
Epoch 6, Step 760 | Training Acc: 0.881 | Test Acc: 0.877 | Test Loss: 0.270 | Test AUC 0.870 (250.68 s) 
(*) Entering Epoch 7 (258.028 s)
Epoch 7, Step 190 | Training Acc: 0.865 | Test Acc: 0.879 | Test Loss: 0.293 | Test AUC 0.860 (264.84 s) 
Epoch 7, Step 380 | Training Acc: 0.875 | Test Acc: 0.895 | Test Loss: 0.300 | Test AUC 0.874 (271.63 s) 
Epoch 7, Step 570 | Training Acc: 0.879 | Test Acc: 0.861 | Test Loss: 0.289 | Test AUC 0.861 (278.41 s) 
Epoch 7, Step 760 | Training Acc: 0.854 | Test Acc: 0.852 | Test Loss: 0.354 | Test AUC 0.832 (285.24 s) 
(*) Entering Epoch 8 (292.623 s)
Epoch 8, Step 190 | Training Acc: 0.889 | Test Acc: 0.893 | Test Loss: 0.262 | Test AUC 0.880 (299.42 s) 
Epoch 8, Step 380 | Training Acc: 0.908 | Test Acc: 0.816 | Test Loss: 0.405 | Test AUC 0.837 (306.21 s) 
Epoch 8, Step 570 | Training Acc: 0.879 | Test Acc: 0.848 | Test Loss: 0.335 | Test AUC 0.845 (312.99 s) 
Epoch 8, Step 760 | Training Acc: 0.859 | Test Acc: 0.879 | Test Loss: 0.266 | Test AUC 0.860 (319.77 s) 
(*) Entering Epoch 9 (327.108 s)
Epoch 9, Step 190 | Training Acc: 0.891 | Test Acc: 0.885 | Test Loss: 0.279 | Test AUC 0.879 (334.87 s) [*]
Epoch 9, Step 380 | Training Acc: 0.879 | Test Acc: 0.871 | Test Loss: 0.320 | Test AUC 0.868 (341.67 s) 
Epoch 9, Step 570 | Training Acc: 0.891 | Test Acc: 0.873 | Test Loss: 0.293 | Test AUC 0.876 (348.47 s) 
Epoch 9, Step 760 | Training Acc: 0.855 | Test Acc: 0.857 | Test Loss: 0.307 | Test AUC 0.844 (355.25 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch10.ckpt-10
(*) Entering Epoch 10 (363.576 s)
Epoch 10, Step 190 | Training Acc: 0.893 | Test Acc: 0.895 | Test Loss: 0.295 | Test AUC 0.877 (371.51 s) [*]
Epoch 10, Step 380 | Training Acc: 0.875 | Test Acc: 0.832 | Test Loss: 0.396 | Test AUC 0.823 (378.33 s) 
Epoch 10, Step 570 | Training Acc: 0.883 | Test Acc: 0.852 | Test Loss: 0.360 | Test AUC 0.855 (385.13 s) 
Epoch 10, Step 760 | Training Acc: 0.877 | Test Acc: 0.887 | Test Loss: 0.284 | Test AUC 0.871 (391.92 s) 
(*) Entering Epoch 11 (399.319 s)
Epoch 11, Step 190 | Training Acc: 0.904 | Test Acc: 0.904 | Test Loss: 0.243 | Test AUC 0.894 (407.07 s) [*]
Epoch 11, Step 380 | Training Acc: 0.885 | Test Acc: 0.881 | Test Loss: 0.275 | Test AUC 0.879 (413.80 s) 
Epoch 11, Step 570 | Training Acc: 0.893 | Test Acc: 0.871 | Test Loss: 0.336 | Test AUC 0.873 (420.56 s) 
Epoch 11, Step 760 | Training Acc: 0.887 | Test Acc: 0.881 | Test Loss: 0.268 | Test AUC 0.874 (427.43 s) 
(*) Entering Epoch 12 (434.786 s)
Epoch 12, Step 190 | Training Acc: 0.879 | Test Acc: 0.891 | Test Loss: 0.280 | Test AUC 0.879 (441.58 s) 
Epoch 12, Step 380 | Training Acc: 0.895 | Test Acc: 0.855 | Test Loss: 0.297 | Test AUC 0.861 (448.34 s) 
Epoch 12, Step 570 | Training Acc: 0.861 | Test Acc: 0.881 | Test Loss: 0.292 | Test AUC 0.859 (455.08 s) 
Epoch 12, Step 760 | Training Acc: 0.910 | Test Acc: 0.865 | Test Loss: 0.287 | Test AUC 0.876 (461.80 s) 
(*) Entering Epoch 13 (469.127 s)
Epoch 13, Step 190 | Training Acc: 0.900 | Test Acc: 0.785 | Test Loss: 0.446 | Test AUC 0.813 (475.88 s) 
Epoch 13, Step 380 | Training Acc: 0.881 | Test Acc: 0.840 | Test Loss: 0.382 | Test AUC 0.841 (482.64 s) 
Epoch 13, Step 570 | Training Acc: 0.875 | Test Acc: 0.869 | Test Loss: 0.308 | Test AUC 0.859 (489.38 s) 
Epoch 13, Step 760 | Training Acc: 0.902 | Test Acc: 0.877 | Test Loss: 0.260 | Test AUC 0.882 (496.15 s) 
(*) Entering Epoch 14 (503.482 s)
Epoch 14, Step 190 | Training Acc: 0.883 | Test Acc: 0.869 | Test Loss: 0.317 | Test AUC 0.867 (510.25 s) 
Epoch 14, Step 380 | Training Acc: 0.904 | Test Acc: 0.906 | Test Loss: 0.253 | Test AUC 0.898 (517.99 s) [*]
Epoch 14, Step 570 | Training Acc: 0.900 | Test Acc: 0.893 | Test Loss: 0.259 | Test AUC 0.885 (524.75 s) 
Epoch 14, Step 760 | Training Acc: 0.879 | Test Acc: 0.885 | Test Loss: 0.284 | Test AUC 0.875 (531.52 s) 
(*) Entering Epoch 15 (538.837 s)
Epoch 15, Step 190 | Training Acc: 0.873 | Test Acc: 0.895 | Test Loss: 0.268 | Test AUC 0.877 (545.57 s) 
Epoch 15, Step 380 | Training Acc: 0.895 | Test Acc: 0.863 | Test Loss: 0.281 | Test AUC 0.860 (552.31 s) 
Epoch 15, Step 570 | Training Acc: 0.895 | Test Acc: 0.895 | Test Loss: 0.264 | Test AUC 0.887 (559.04 s) 
Epoch 15, Step 760 | Training Acc: 0.891 | Test Acc: 0.891 | Test Loss: 0.278 | Test AUC 0.877 (565.77 s) 
(*) Entering Epoch 16 (573.085 s)
Epoch 16, Step 190 | Training Acc: 0.883 | Test Acc: 0.900 | Test Loss: 0.257 | Test AUC 0.881 (579.83 s) 
Epoch 16, Step 380 | Training Acc: 0.879 | Test Acc: 0.885 | Test Loss: 0.270 | Test AUC 0.869 (586.56 s) 
Epoch 16, Step 570 | Training Acc: 0.877 | Test Acc: 0.873 | Test Loss: 0.290 | Test AUC 0.868 (593.30 s) 
Epoch 16, Step 760 | Training Acc: 0.871 | Test Acc: 0.857 | Test Loss: 0.326 | Test AUC 0.850 (600.03 s) 
(*) Entering Epoch 17 (607.352 s)
Epoch 17, Step 190 | Training Acc: 0.889 | Test Acc: 0.879 | Test Loss: 0.282 | Test AUC 0.869 (614.13 s) 
Epoch 17, Step 380 | Training Acc: 0.879 | Test Acc: 0.873 | Test Loss: 0.309 | Test AUC 0.856 (620.87 s) 
Epoch 17, Step 570 | Training Acc: 0.869 | Test Acc: 0.883 | Test Loss: 0.291 | Test AUC 0.869 (627.63 s) 
Epoch 17, Step 760 | Training Acc: 0.889 | Test Acc: 0.805 | Test Loss: 0.465 | Test AUC 0.818 (634.41 s) 
(*) Entering Epoch 18 (641.769 s)
Epoch 18, Step 190 | Training Acc: 0.895 | Test Acc: 0.816 | Test Loss: 0.430 | Test AUC 0.820 (648.51 s) 
Epoch 18, Step 380 | Training Acc: 0.896 | Test Acc: 0.883 | Test Loss: 0.304 | Test AUC 0.874 (655.26 s) 
Epoch 18, Step 570 | Training Acc: 0.895 | Test Acc: 0.865 | Test Loss: 0.304 | Test AUC 0.871 (661.99 s) 
Epoch 18, Step 760 | Training Acc: 0.852 | Test Acc: 0.873 | Test Loss: 0.285 | Test AUC 0.852 (668.71 s) 
(*) Entering Epoch 19 (676.046 s)
Epoch 19, Step 190 | Training Acc: 0.908 | Test Acc: 0.811 | Test Loss: 0.405 | Test AUC 0.835 (682.78 s) 
Epoch 19, Step 380 | Training Acc: 0.898 | Test Acc: 0.881 | Test Loss: 0.294 | Test AUC 0.881 (689.51 s) 
Epoch 19, Step 570 | Training Acc: 0.895 | Test Acc: 0.893 | Test Loss: 0.250 | Test AUC 0.881 (696.26 s) 
Epoch 19, Step 760 | Training Acc: 0.906 | Test Acc: 0.875 | Test Loss: 0.301 | Test AUC 0.880 (703.04 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch20.ckpt-20
(*) Entering Epoch 20 (711.323 s)
Epoch 20, Step 190 | Training Acc: 0.900 | Test Acc: 0.861 | Test Loss: 0.331 | Test AUC 0.868 (718.06 s) 
Epoch 20, Step 380 | Training Acc: 0.885 | Test Acc: 0.838 | Test Loss: 0.397 | Test AUC 0.839 (724.81 s) 
Epoch 20, Step 570 | Training Acc: 0.891 | Test Acc: 0.826 | Test Loss: 0.366 | Test AUC 0.838 (731.54 s) 
Epoch 20, Step 760 | Training Acc: 0.891 | Test Acc: 0.885 | Test Loss: 0.300 | Test AUC 0.876 (738.28 s) 
(*) Entering Epoch 21 (745.591 s)
Epoch 21, Step 190 | Training Acc: 0.900 | Test Acc: 0.797 | Test Loss: 0.490 | Test AUC 0.824 (752.33 s) 
Epoch 21, Step 380 | Training Acc: 0.881 | Test Acc: 0.861 | Test Loss: 0.331 | Test AUC 0.852 (759.08 s) 
Epoch 21, Step 570 | Training Acc: 0.908 | Test Acc: 0.885 | Test Loss: 0.311 | Test AUC 0.885 (765.85 s) 
Epoch 21, Step 760 | Training Acc: 0.893 | Test Acc: 0.869 | Test Loss: 0.306 | Test AUC 0.872 (772.56 s) 
(*) Entering Epoch 22 (779.910 s)
Epoch 22, Step 190 | Training Acc: 0.904 | Test Acc: 0.877 | Test Loss: 0.301 | Test AUC 0.878 (786.66 s) 
Epoch 22, Step 380 | Training Acc: 0.863 | Test Acc: 0.904 | Test Loss: 0.239 | Test AUC 0.869 (793.41 s) 
Epoch 22, Step 570 | Training Acc: 0.904 | Test Acc: 0.859 | Test Loss: 0.348 | Test AUC 0.865 (800.13 s) 
Epoch 22, Step 760 | Training Acc: 0.920 | Test Acc: 0.889 | Test Loss: 0.286 | Test AUC 0.895 (806.88 s) 
(*) Entering Epoch 23 (814.238 s)
Epoch 23, Step 190 | Training Acc: 0.898 | Test Acc: 0.793 | Test Loss: 0.424 | Test AUC 0.803 (820.97 s) 
Epoch 23, Step 380 | Training Acc: 0.895 | Test Acc: 0.818 | Test Loss: 0.432 | Test AUC 0.837 (827.70 s) 
Epoch 23, Step 570 | Training Acc: 0.922 | Test Acc: 0.783 | Test Loss: 0.494 | Test AUC 0.819 (834.43 s) 
Epoch 23, Step 760 | Training Acc: 0.891 | Test Acc: 0.811 | Test Loss: 0.404 | Test AUC 0.831 (841.19 s) 
(*) Entering Epoch 24 (848.515 s)
Epoch 24, Step 190 | Training Acc: 0.906 | Test Acc: 0.889 | Test Loss: 0.291 | Test AUC 0.883 (855.24 s) 
Epoch 24, Step 380 | Training Acc: 0.928 | Test Acc: 0.881 | Test Loss: 0.280 | Test AUC 0.894 (862.03 s) 
Epoch 24, Step 570 | Training Acc: 0.918 | Test Acc: 0.873 | Test Loss: 0.298 | Test AUC 0.888 (868.77 s) 
Epoch 24, Step 760 | Training Acc: 0.922 | Test Acc: 0.875 | Test Loss: 0.328 | Test AUC 0.888 (875.51 s) 
(*) Entering Epoch 25 (882.841 s)
Epoch 25, Step 190 | Training Acc: 0.898 | Test Acc: 0.871 | Test Loss: 0.280 | Test AUC 0.870 (889.57 s) 
Epoch 25, Step 380 | Training Acc: 0.902 | Test Acc: 0.883 | Test Loss: 0.309 | Test AUC 0.886 (896.30 s) 
Epoch 25, Step 570 | Training Acc: 0.896 | Test Acc: 0.801 | Test Loss: 0.507 | Test AUC 0.822 (903.07 s) 
Epoch 25, Step 760 | Training Acc: 0.893 | Test Acc: 0.848 | Test Loss: 0.347 | Test AUC 0.851 (909.79 s) 
(*) Entering Epoch 26 (917.133 s)
Epoch 26, Step 190 | Training Acc: 0.891 | Test Acc: 0.785 | Test Loss: 0.527 | Test AUC 0.803 (923.90 s) 
Epoch 26, Step 380 | Training Acc: 0.902 | Test Acc: 0.850 | Test Loss: 0.373 | Test AUC 0.860 (930.63 s) 
Epoch 26, Step 570 | Training Acc: 0.887 | Test Acc: 0.891 | Test Loss: 0.296 | Test AUC 0.876 (937.38 s) 
Epoch 26, Step 760 | Training Acc: 0.881 | Test Acc: 0.799 | Test Loss: 0.468 | Test AUC 0.813 (944.12 s) 
(*) Entering Epoch 27 (951.452 s)
Epoch 27, Step 190 | Training Acc: 0.908 | Test Acc: 0.793 | Test Loss: 0.418 | Test AUC 0.825 (958.21 s) 
Epoch 27, Step 380 | Training Acc: 0.904 | Test Acc: 0.822 | Test Loss: 0.429 | Test AUC 0.833 (964.96 s) 
Epoch 27, Step 570 | Training Acc: 0.891 | Test Acc: 0.836 | Test Loss: 0.395 | Test AUC 0.838 (971.73 s) 
Epoch 27, Step 760 | Training Acc: 0.922 | Test Acc: 0.859 | Test Loss: 0.322 | Test AUC 0.877 (978.53 s) 
(*) Entering Epoch 28 (985.882 s)
Epoch 28, Step 190 | Training Acc: 0.908 | Test Acc: 0.895 | Test Loss: 0.284 | Test AUC 0.889 (992.63 s) 
Epoch 28, Step 380 | Training Acc: 0.891 | Test Acc: 0.859 | Test Loss: 0.312 | Test AUC 0.863 (999.38 s) 
Epoch 28, Step 570 | Training Acc: 0.916 | Test Acc: 0.885 | Test Loss: 0.317 | Test AUC 0.895 (1006.11 s) 
Epoch 28, Step 760 | Training Acc: 0.908 | Test Acc: 0.826 | Test Loss: 0.385 | Test AUC 0.849 (1012.87 s) 
(*) Entering Epoch 29 (1020.219 s)
Epoch 29, Step 190 | Training Acc: 0.904 | Test Acc: 0.904 | Test Loss: 0.281 | Test AUC 0.895 (1026.97 s) 
Epoch 29, Step 380 | Training Acc: 0.908 | Test Acc: 0.912 | Test Loss: 0.225 | Test AUC 0.902 (1034.68 s) [*]
Epoch 29, Step 570 | Training Acc: 0.902 | Test Acc: 0.885 | Test Loss: 0.302 | Test AUC 0.886 (1041.42 s) 
Epoch 29, Step 760 | Training Acc: 0.887 | Test Acc: 0.879 | Test Loss: 0.291 | Test AUC 0.876 (1048.19 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch30.ckpt-30
(*) Entering Epoch 30 (1056.443 s)
Epoch 30, Step 190 | Training Acc: 0.920 | Test Acc: 0.803 | Test Loss: 0.479 | Test AUC 0.837 (1063.20 s) 
Epoch 30, Step 380 | Training Acc: 0.914 | Test Acc: 0.879 | Test Loss: 0.315 | Test AUC 0.886 (1069.96 s) 
Epoch 30, Step 570 | Training Acc: 0.908 | Test Acc: 0.779 | Test Loss: 0.461 | Test AUC 0.816 (1076.69 s) 
Epoch 30, Step 760 | Training Acc: 0.918 | Test Acc: 0.799 | Test Loss: 0.442 | Test AUC 0.835 (1083.43 s) 
(*) Entering Epoch 31 (1090.772 s)
Epoch 31, Step 190 | Training Acc: 0.902 | Test Acc: 0.854 | Test Loss: 0.321 | Test AUC 0.868 (1097.52 s) 
Epoch 31, Step 380 | Training Acc: 0.926 | Test Acc: 0.865 | Test Loss: 0.297 | Test AUC 0.882 (1104.26 s) 
Epoch 31, Step 570 | Training Acc: 0.916 | Test Acc: 0.785 | Test Loss: 0.491 | Test AUC 0.824 (1111.00 s) 
Epoch 31, Step 760 | Training Acc: 0.881 | Test Acc: 0.844 | Test Loss: 0.366 | Test AUC 0.842 (1117.73 s) 
(*) Entering Epoch 32 (1125.067 s)
Epoch 32, Step 190 | Training Acc: 0.883 | Test Acc: 0.898 | Test Loss: 0.256 | Test AUC 0.877 (1131.82 s) 
Epoch 32, Step 380 | Training Acc: 0.930 | Test Acc: 0.854 | Test Loss: 0.332 | Test AUC 0.884 (1138.56 s) 
Epoch 32, Step 570 | Training Acc: 0.916 | Test Acc: 0.885 | Test Loss: 0.273 | Test AUC 0.891 (1145.28 s) 
Epoch 32, Step 760 | Training Acc: 0.904 | Test Acc: 0.883 | Test Loss: 0.268 | Test AUC 0.883 (1152.03 s) 
(*) Entering Epoch 33 (1159.367 s)
Epoch 33, Step 190 | Training Acc: 0.908 | Test Acc: 0.834 | Test Loss: 0.433 | Test AUC 0.850 (1166.10 s) 
Epoch 33, Step 380 | Training Acc: 0.920 | Test Acc: 0.881 | Test Loss: 0.342 | Test AUC 0.893 (1172.84 s) 
Epoch 33, Step 570 | Training Acc: 0.891 | Test Acc: 0.863 | Test Loss: 0.334 | Test AUC 0.861 (1179.59 s) 
Epoch 33, Step 760 | Training Acc: 0.895 | Test Acc: 0.904 | Test Loss: 0.229 | Test AUC 0.888 (1186.32 s) 
(*) Entering Epoch 34 (1193.652 s)
Epoch 34, Step 190 | Training Acc: 0.926 | Test Acc: 0.889 | Test Loss: 0.286 | Test AUC 0.902 (1200.41 s) 
Epoch 34, Step 380 | Training Acc: 0.906 | Test Acc: 0.879 | Test Loss: 0.266 | Test AUC 0.880 (1207.14 s) 
Epoch 34, Step 570 | Training Acc: 0.896 | Test Acc: 0.875 | Test Loss: 0.315 | Test AUC 0.877 (1213.90 s) 
Epoch 34, Step 760 | Training Acc: 0.910 | Test Acc: 0.852 | Test Loss: 0.368 | Test AUC 0.870 (1220.63 s) 
(*) Entering Epoch 35 (1227.974 s)
Epoch 35, Step 190 | Training Acc: 0.906 | Test Acc: 0.887 | Test Loss: 0.296 | Test AUC 0.889 (1234.71 s) 
Epoch 35, Step 380 | Training Acc: 0.902 | Test Acc: 0.902 | Test Loss: 0.243 | Test AUC 0.896 (1241.46 s) 
Epoch 35, Step 570 | Training Acc: 0.910 | Test Acc: 0.820 | Test Loss: 0.416 | Test AUC 0.849 (1248.19 s) 
Epoch 35, Step 760 | Training Acc: 0.908 | Test Acc: 0.879 | Test Loss: 0.304 | Test AUC 0.882 (1254.92 s) 
(*) Entering Epoch 36 (1262.254 s)
Epoch 36, Step 190 | Training Acc: 0.900 | Test Acc: 0.881 | Test Loss: 0.271 | Test AUC 0.886 (1269.00 s) 
Epoch 36, Step 380 | Training Acc: 0.904 | Test Acc: 0.908 | Test Loss: 0.232 | Test AUC 0.898 (1275.75 s) 
Epoch 36, Step 570 | Training Acc: 0.910 | Test Acc: 0.893 | Test Loss: 0.302 | Test AUC 0.893 (1282.55 s) 
Epoch 36, Step 760 | Training Acc: 0.885 | Test Acc: 0.879 | Test Loss: 0.313 | Test AUC 0.874 (1289.27 s) 
(*) Entering Epoch 37 (1296.624 s)
Epoch 37, Step 190 | Training Acc: 0.902 | Test Acc: 0.836 | Test Loss: 0.389 | Test AUC 0.843 (1303.35 s) 
Epoch 37, Step 380 | Training Acc: 0.902 | Test Acc: 0.879 | Test Loss: 0.304 | Test AUC 0.886 (1310.09 s) 
Epoch 37, Step 570 | Training Acc: 0.893 | Test Acc: 0.867 | Test Loss: 0.302 | Test AUC 0.861 (1316.86 s) 
Epoch 37, Step 760 | Training Acc: 0.912 | Test Acc: 0.869 | Test Loss: 0.322 | Test AUC 0.884 (1323.62 s) 
(*) Entering Epoch 38 (1330.940 s)
Epoch 38, Step 190 | Training Acc: 0.896 | Test Acc: 0.879 | Test Loss: 0.325 | Test AUC 0.878 (1337.68 s) 
Epoch 38, Step 380 | Training Acc: 0.906 | Test Acc: 0.898 | Test Loss: 0.258 | Test AUC 0.894 (1344.44 s) 
Epoch 38, Step 570 | Training Acc: 0.906 | Test Acc: 0.820 | Test Loss: 0.390 | Test AUC 0.837 (1351.19 s) 
Epoch 38, Step 760 | Training Acc: 0.893 | Test Acc: 0.824 | Test Loss: 0.386 | Test AUC 0.839 (1357.93 s) 
(*) Entering Epoch 39 (1365.285 s)
Epoch 39, Step 190 | Training Acc: 0.912 | Test Acc: 0.865 | Test Loss: 0.314 | Test AUC 0.878 (1372.02 s) 
Epoch 39, Step 380 | Training Acc: 0.885 | Test Acc: 0.754 | Test Loss: 0.566 | Test AUC 0.788 (1378.78 s) 
Epoch 39, Step 570 | Training Acc: 0.893 | Test Acc: 0.850 | Test Loss: 0.340 | Test AUC 0.851 (1385.52 s) 
Epoch 39, Step 760 | Training Acc: 0.930 | Test Acc: 0.885 | Test Loss: 0.303 | Test AUC 0.896 (1392.29 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch40.ckpt-40
(*) Entering Epoch 40 (1400.586 s)
Epoch 40, Step 190 | Training Acc: 0.922 | Test Acc: 0.877 | Test Loss: 0.297 | Test AUC 0.883 (1407.33 s) 
Epoch 40, Step 380 | Training Acc: 0.865 | Test Acc: 0.873 | Test Loss: 0.311 | Test AUC 0.855 (1414.07 s) 
Epoch 40, Step 570 | Training Acc: 0.928 | Test Acc: 0.846 | Test Loss: 0.373 | Test AUC 0.879 (1420.82 s) 
Epoch 40, Step 760 | Training Acc: 0.908 | Test Acc: 0.896 | Test Loss: 0.251 | Test AUC 0.896 (1427.58 s) 
(*) Entering Epoch 41 (1434.895 s)
Epoch 41, Step 190 | Training Acc: 0.895 | Test Acc: 0.809 | Test Loss: 0.467 | Test AUC 0.836 (1441.65 s) 
Epoch 41, Step 380 | Training Acc: 0.883 | Test Acc: 0.861 | Test Loss: 0.359 | Test AUC 0.860 (1448.38 s) 
Epoch 41, Step 570 | Training Acc: 0.908 | Test Acc: 0.867 | Test Loss: 0.299 | Test AUC 0.876 (1455.14 s) 
Epoch 41, Step 760 | Training Acc: 0.904 | Test Acc: 0.902 | Test Loss: 0.229 | Test AUC 0.895 (1461.88 s) 
(*) Entering Epoch 42 (1469.246 s)
Epoch 42, Step 190 | Training Acc: 0.914 | Test Acc: 0.824 | Test Loss: 0.480 | Test AUC 0.852 (1475.99 s) 
Epoch 42, Step 380 | Training Acc: 0.900 | Test Acc: 0.822 | Test Loss: 0.436 | Test AUC 0.835 (1482.72 s) 
Epoch 42, Step 570 | Training Acc: 0.904 | Test Acc: 0.865 | Test Loss: 0.337 | Test AUC 0.870 (1489.48 s) 
Epoch 42, Step 760 | Training Acc: 0.908 | Test Acc: 0.883 | Test Loss: 0.297 | Test AUC 0.886 (1496.22 s) 
(*) Entering Epoch 43 (1503.560 s)
Epoch 43, Step 190 | Training Acc: 0.898 | Test Acc: 0.902 | Test Loss: 0.253 | Test AUC 0.889 (1510.30 s) 
Epoch 43, Step 380 | Training Acc: 0.928 | Test Acc: 0.889 | Test Loss: 0.270 | Test AUC 0.903 (1517.05 s) 
Epoch 43, Step 570 | Training Acc: 0.904 | Test Acc: 0.885 | Test Loss: 0.269 | Test AUC 0.888 (1523.79 s) 
Epoch 43, Step 760 | Training Acc: 0.912 | Test Acc: 0.867 | Test Loss: 0.319 | Test AUC 0.880 (1530.55 s) 
(*) Entering Epoch 44 (1537.875 s)
Epoch 44, Step 190 | Training Acc: 0.928 | Test Acc: 0.865 | Test Loss: 0.312 | Test AUC 0.891 (1544.62 s) 
Epoch 44, Step 380 | Training Acc: 0.904 | Test Acc: 0.883 | Test Loss: 0.267 | Test AUC 0.882 (1551.38 s) 
Epoch 44, Step 570 | Training Acc: 0.924 | Test Acc: 0.887 | Test Loss: 0.261 | Test AUC 0.901 (1558.13 s) 
Epoch 44, Step 760 | Training Acc: 0.928 | Test Acc: 0.879 | Test Loss: 0.318 | Test AUC 0.891 (1564.88 s) 
(*) Entering Epoch 45 (1572.229 s)
Epoch 45, Step 190 | Training Acc: 0.914 | Test Acc: 0.881 | Test Loss: 0.259 | Test AUC 0.882 (1578.98 s) 
Epoch 45, Step 380 | Training Acc: 0.889 | Test Acc: 0.857 | Test Loss: 0.352 | Test AUC 0.858 (1585.73 s) 
Epoch 45, Step 570 | Training Acc: 0.893 | Test Acc: 0.814 | Test Loss: 0.409 | Test AUC 0.831 (1592.50 s) 
Epoch 45, Step 760 | Training Acc: 0.904 | Test Acc: 0.820 | Test Loss: 0.407 | Test AUC 0.843 (1599.24 s) 
(*) Entering Epoch 46 (1606.572 s)
Epoch 46, Step 190 | Training Acc: 0.910 | Test Acc: 0.871 | Test Loss: 0.292 | Test AUC 0.880 (1613.32 s) 
Epoch 46, Step 380 | Training Acc: 0.922 | Test Acc: 0.877 | Test Loss: 0.296 | Test AUC 0.889 (1620.06 s) 
Epoch 46, Step 570 | Training Acc: 0.926 | Test Acc: 0.861 | Test Loss: 0.307 | Test AUC 0.886 (1626.78 s) 
Epoch 46, Step 760 | Training Acc: 0.906 | Test Acc: 0.752 | Test Loss: 0.540 | Test AUC 0.796 (1633.53 s) 
(*) Entering Epoch 47 (1640.858 s)
Epoch 47, Step 190 | Training Acc: 0.914 | Test Acc: 0.820 | Test Loss: 0.484 | Test AUC 0.841 (1647.62 s) 
Epoch 47, Step 380 | Training Acc: 0.926 | Test Acc: 0.801 | Test Loss: 0.439 | Test AUC 0.844 (1654.36 s) 
Epoch 47, Step 570 | Training Acc: 0.908 | Test Acc: 0.842 | Test Loss: 0.341 | Test AUC 0.858 (1661.12 s) 
Epoch 47, Step 760 | Training Acc: 0.916 | Test Acc: 0.859 | Test Loss: 0.326 | Test AUC 0.876 (1667.87 s) 
(*) Entering Epoch 48 (1675.207 s)
Epoch 48, Step 190 | Training Acc: 0.924 | Test Acc: 0.857 | Test Loss: 0.328 | Test AUC 0.879 (1681.98 s) 
Epoch 48, Step 380 | Training Acc: 0.926 | Test Acc: 0.879 | Test Loss: 0.323 | Test AUC 0.896 (1688.72 s) 
Epoch 48, Step 570 | Training Acc: 0.930 | Test Acc: 0.885 | Test Loss: 0.288 | Test AUC 0.902 (1695.45 s) 
Epoch 48, Step 760 | Training Acc: 0.906 | Test Acc: 0.854 | Test Loss: 0.365 | Test AUC 0.867 (1702.27 s) 
(*) Entering Epoch 49 (1709.591 s)
Epoch 49, Step 190 | Training Acc: 0.889 | Test Acc: 0.854 | Test Loss: 0.340 | Test AUC 0.850 (1716.33 s) 
Epoch 49, Step 380 | Training Acc: 0.918 | Test Acc: 0.895 | Test Loss: 0.261 | Test AUC 0.895 (1723.07 s) 
Epoch 49, Step 570 | Training Acc: 0.900 | Test Acc: 0.766 | Test Loss: 0.582 | Test AUC 0.802 (1729.82 s) 
Epoch 49, Step 760 | Training Acc: 0.920 | Test Acc: 0.850 | Test Loss: 0.397 | Test AUC 0.874 (1736.57 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch50.ckpt-50
(*) Entering Epoch 50 (1744.951 s)
Epoch 50, Step 190 | Training Acc: 0.922 | Test Acc: 0.871 | Test Loss: 0.327 | Test AUC 0.888 (1751.75 s) 
Epoch 50, Step 380 | Training Acc: 0.930 | Test Acc: 0.891 | Test Loss: 0.273 | Test AUC 0.899 (1758.51 s) 
Epoch 50, Step 570 | Training Acc: 0.908 | Test Acc: 0.891 | Test Loss: 0.253 | Test AUC 0.890 (1765.26 s) 
Epoch 50, Step 760 | Training Acc: 0.920 | Test Acc: 0.895 | Test Loss: 0.269 | Test AUC 0.903 (1772.01 s) 
(*) Entering Epoch 51 (1779.312 s)
Epoch 51, Step 190 | Training Acc: 0.918 | Test Acc: 0.883 | Test Loss: 0.288 | Test AUC 0.893 (1786.05 s) 
Epoch 51, Step 380 | Training Acc: 0.916 | Test Acc: 0.795 | Test Loss: 0.477 | Test AUC 0.834 (1792.80 s) 
Epoch 51, Step 570 | Training Acc: 0.902 | Test Acc: 0.789 | Test Loss: 0.507 | Test AUC 0.819 (1799.55 s) 
Epoch 51, Step 760 | Training Acc: 0.885 | Test Acc: 0.777 | Test Loss: 0.489 | Test AUC 0.799 (1806.29 s) 
(*) Entering Epoch 52 (1813.656 s)
Epoch 52, Step 190 | Training Acc: 0.914 | Test Acc: 0.857 | Test Loss: 0.323 | Test AUC 0.873 (1820.41 s) 
Epoch 52, Step 380 | Training Acc: 0.918 | Test Acc: 0.770 | Test Loss: 0.613 | Test AUC 0.800 (1827.17 s) 
Epoch 52, Step 570 | Training Acc: 0.926 | Test Acc: 0.795 | Test Loss: 0.451 | Test AUC 0.844 (1833.92 s) 
Epoch 52, Step 760 | Training Acc: 0.912 | Test Acc: 0.820 | Test Loss: 0.460 | Test AUC 0.848 (1840.67 s) 
(*) Entering Epoch 53 (1847.992 s)
Epoch 53, Step 190 | Training Acc: 0.926 | Test Acc: 0.871 | Test Loss: 0.314 | Test AUC 0.887 (1854.71 s) 
Epoch 53, Step 380 | Training Acc: 0.918 | Test Acc: 0.902 | Test Loss: 0.270 | Test AUC 0.900 (1861.45 s) 
Epoch 53, Step 570 | Training Acc: 0.924 | Test Acc: 0.916 | Test Loss: 0.257 | Test AUC 0.913 (1869.17 s) [*]
Epoch 53, Step 760 | Training Acc: 0.922 | Test Acc: 0.896 | Test Loss: 0.270 | Test AUC 0.900 (1875.92 s) 
(*) Entering Epoch 54 (1883.230 s)
Epoch 54, Step 190 | Training Acc: 0.934 | Test Acc: 0.875 | Test Loss: 0.308 | Test AUC 0.893 (1889.98 s) 
Epoch 54, Step 380 | Training Acc: 0.928 | Test Acc: 0.885 | Test Loss: 0.276 | Test AUC 0.890 (1896.73 s) 
Epoch 54, Step 570 | Training Acc: 0.906 | Test Acc: 0.889 | Test Loss: 0.254 | Test AUC 0.892 (1903.47 s) 
Epoch 54, Step 760 | Training Acc: 0.926 | Test Acc: 0.885 | Test Loss: 0.291 | Test AUC 0.897 (1910.22 s) 
(*) Entering Epoch 55 (1917.534 s)
Epoch 55, Step 190 | Training Acc: 0.922 | Test Acc: 0.877 | Test Loss: 0.314 | Test AUC 0.891 (1924.29 s) 
Epoch 55, Step 380 | Training Acc: 0.936 | Test Acc: 0.883 | Test Loss: 0.285 | Test AUC 0.903 (1931.05 s) 
Epoch 55, Step 570 | Training Acc: 0.920 | Test Acc: 0.814 | Test Loss: 0.455 | Test AUC 0.844 (1937.78 s) 
Epoch 55, Step 760 | Training Acc: 0.910 | Test Acc: 0.775 | Test Loss: 0.561 | Test AUC 0.822 (1944.52 s) 
(*) Entering Epoch 56 (1951.871 s)
Epoch 56, Step 190 | Training Acc: 0.936 | Test Acc: 0.822 | Test Loss: 0.403 | Test AUC 0.862 (1958.61 s) 
Epoch 56, Step 380 | Training Acc: 0.904 | Test Acc: 0.877 | Test Loss: 0.313 | Test AUC 0.882 (1965.36 s) 
Epoch 56, Step 570 | Training Acc: 0.918 | Test Acc: 0.879 | Test Loss: 0.315 | Test AUC 0.889 (1972.10 s) 
Epoch 56, Step 760 | Training Acc: 0.928 | Test Acc: 0.857 | Test Loss: 0.334 | Test AUC 0.887 (1978.84 s) 
(*) Entering Epoch 57 (1986.161 s)
Epoch 57, Step 190 | Training Acc: 0.934 | Test Acc: 0.893 | Test Loss: 0.283 | Test AUC 0.909 (1992.93 s) 
Epoch 57, Step 380 | Training Acc: 0.920 | Test Acc: 0.869 | Test Loss: 0.334 | Test AUC 0.883 (1999.64 s) 
Epoch 57, Step 570 | Training Acc: 0.918 | Test Acc: 0.885 | Test Loss: 0.316 | Test AUC 0.889 (2006.38 s) 
Epoch 57, Step 760 | Training Acc: 0.922 | Test Acc: 0.863 | Test Loss: 0.330 | Test AUC 0.886 (2013.13 s) 
(*) Entering Epoch 58 (2020.442 s)
Epoch 58, Step 190 | Training Acc: 0.930 | Test Acc: 0.803 | Test Loss: 0.467 | Test AUC 0.847 (2027.17 s) 
Epoch 58, Step 380 | Training Acc: 0.902 | Test Acc: 0.848 | Test Loss: 0.365 | Test AUC 0.865 (2033.91 s) 
Epoch 58, Step 570 | Training Acc: 0.912 | Test Acc: 0.881 | Test Loss: 0.312 | Test AUC 0.884 (2040.68 s) 
Epoch 58, Step 760 | Training Acc: 0.912 | Test Acc: 0.877 | Test Loss: 0.272 | Test AUC 0.885 (2047.40 s) 
(*) Entering Epoch 59 (2054.762 s)
Epoch 59, Step 190 | Training Acc: 0.906 | Test Acc: 0.795 | Test Loss: 0.496 | Test AUC 0.821 (2061.55 s) 
Epoch 59, Step 380 | Training Acc: 0.918 | Test Acc: 0.855 | Test Loss: 0.418 | Test AUC 0.871 (2068.28 s) 
Epoch 59, Step 570 | Training Acc: 0.922 | Test Acc: 0.861 | Test Loss: 0.348 | Test AUC 0.878 (2075.04 s) 
Epoch 59, Step 760 | Training Acc: 0.934 | Test Acc: 0.811 | Test Loss: 0.460 | Test AUC 0.846 (2081.78 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch60.ckpt-60
(*) Entering Epoch 60 (2090.163 s)
Epoch 60, Step 190 | Training Acc: 0.932 | Test Acc: 0.857 | Test Loss: 0.355 | Test AUC 0.884 (2096.90 s) 
Epoch 60, Step 380 | Training Acc: 0.916 | Test Acc: 0.799 | Test Loss: 0.494 | Test AUC 0.836 (2103.67 s) 
Epoch 60, Step 570 | Training Acc: 0.900 | Test Acc: 0.879 | Test Loss: 0.291 | Test AUC 0.879 (2110.40 s) 
Epoch 60, Step 760 | Training Acc: 0.930 | Test Acc: 0.877 | Test Loss: 0.315 | Test AUC 0.896 (2117.16 s) 
(*) Entering Epoch 61 (2124.483 s)
Epoch 61, Step 190 | Training Acc: 0.912 | Test Acc: 0.830 | Test Loss: 0.446 | Test AUC 0.855 (2131.28 s) 
Epoch 61, Step 380 | Training Acc: 0.922 | Test Acc: 0.838 | Test Loss: 0.411 | Test AUC 0.866 (2138.03 s) 
Epoch 61, Step 570 | Training Acc: 0.922 | Test Acc: 0.855 | Test Loss: 0.347 | Test AUC 0.881 (2144.81 s) 
Epoch 61, Step 760 | Training Acc: 0.916 | Test Acc: 0.889 | Test Loss: 0.287 | Test AUC 0.896 (2151.57 s) 
(*) Entering Epoch 62 (2158.915 s)
Epoch 62, Step 190 | Training Acc: 0.924 | Test Acc: 0.852 | Test Loss: 0.351 | Test AUC 0.881 (2165.66 s) 
Epoch 62, Step 380 | Training Acc: 0.930 | Test Acc: 0.867 | Test Loss: 0.315 | Test AUC 0.891 (2172.41 s) 
Epoch 62, Step 570 | Training Acc: 0.928 | Test Acc: 0.836 | Test Loss: 0.380 | Test AUC 0.866 (2179.17 s) 
Epoch 62, Step 760 | Training Acc: 0.920 | Test Acc: 0.863 | Test Loss: 0.311 | Test AUC 0.883 (2185.92 s) 
(*) Entering Epoch 63 (2193.238 s)
Epoch 63, Step 190 | Training Acc: 0.918 | Test Acc: 0.912 | Test Loss: 0.258 | Test AUC 0.911 (2200.00 s) 
Epoch 63, Step 380 | Training Acc: 0.941 | Test Acc: 0.785 | Test Loss: 0.542 | Test AUC 0.836 (2206.75 s) 
Epoch 63, Step 570 | Training Acc: 0.936 | Test Acc: 0.805 | Test Loss: 0.468 | Test AUC 0.853 (2213.49 s) 
Epoch 63, Step 760 | Training Acc: 0.895 | Test Acc: 0.857 | Test Loss: 0.378 | Test AUC 0.859 (2220.23 s) 
(*) Entering Epoch 64 (2227.562 s)
Epoch 64, Step 190 | Training Acc: 0.928 | Test Acc: 0.816 | Test Loss: 0.461 | Test AUC 0.858 (2234.31 s) 
Epoch 64, Step 380 | Training Acc: 0.908 | Test Acc: 0.824 | Test Loss: 0.472 | Test AUC 0.853 (2241.05 s) 
Epoch 64, Step 570 | Training Acc: 0.910 | Test Acc: 0.873 | Test Loss: 0.302 | Test AUC 0.884 (2247.79 s) 
Epoch 64, Step 760 | Training Acc: 0.922 | Test Acc: 0.904 | Test Loss: 0.272 | Test AUC 0.907 (2254.55 s) 
(*) Entering Epoch 65 (2261.880 s)
Epoch 65, Step 190 | Training Acc: 0.918 | Test Acc: 0.893 | Test Loss: 0.296 | Test AUC 0.902 (2268.63 s) 
Epoch 65, Step 380 | Training Acc: 0.920 | Test Acc: 0.795 | Test Loss: 0.482 | Test AUC 0.834 (2275.37 s) 
Epoch 65, Step 570 | Training Acc: 0.928 | Test Acc: 0.838 | Test Loss: 0.394 | Test AUC 0.871 (2282.11 s) 
Epoch 65, Step 760 | Training Acc: 0.910 | Test Acc: 0.887 | Test Loss: 0.269 | Test AUC 0.886 (2288.87 s) 
(*) Entering Epoch 66 (2296.217 s)
Epoch 66, Step 190 | Training Acc: 0.910 | Test Acc: 0.885 | Test Loss: 0.324 | Test AUC 0.891 (2302.96 s) 
Epoch 66, Step 380 | Training Acc: 0.904 | Test Acc: 0.793 | Test Loss: 0.524 | Test AUC 0.824 (2309.71 s) 
Epoch 66, Step 570 | Training Acc: 0.941 | Test Acc: 0.836 | Test Loss: 0.401 | Test AUC 0.875 (2316.46 s) 
Epoch 66, Step 760 | Training Acc: 0.926 | Test Acc: 0.814 | Test Loss: 0.430 | Test AUC 0.849 (2323.21 s) 
(*) Entering Epoch 67 (2330.584 s)
Epoch 67, Step 190 | Training Acc: 0.924 | Test Acc: 0.885 | Test Loss: 0.284 | Test AUC 0.895 (2337.33 s) 
Epoch 67, Step 380 | Training Acc: 0.930 | Test Acc: 0.883 | Test Loss: 0.325 | Test AUC 0.900 (2344.08 s) 
Epoch 67, Step 570 | Training Acc: 0.918 | Test Acc: 0.863 | Test Loss: 0.306 | Test AUC 0.885 (2350.85 s) 
Epoch 67, Step 760 | Training Acc: 0.914 | Test Acc: 0.898 | Test Loss: 0.278 | Test AUC 0.903 (2357.58 s) 
(*) Entering Epoch 68 (2364.913 s)
Epoch 68, Step 190 | Training Acc: 0.930 | Test Acc: 0.869 | Test Loss: 0.333 | Test AUC 0.889 (2371.66 s) 
Epoch 68, Step 380 | Training Acc: 0.916 | Test Acc: 0.900 | Test Loss: 0.286 | Test AUC 0.900 (2378.43 s) 
Epoch 68, Step 570 | Training Acc: 0.924 | Test Acc: 0.893 | Test Loss: 0.286 | Test AUC 0.904 (2385.19 s) 
Epoch 68, Step 760 | Training Acc: 0.918 | Test Acc: 0.861 | Test Loss: 0.386 | Test AUC 0.883 (2391.92 s) 
(*) Entering Epoch 69 (2399.255 s)
Epoch 69, Step 190 | Training Acc: 0.920 | Test Acc: 0.820 | Test Loss: 0.496 | Test AUC 0.851 (2406.02 s) 
Epoch 69, Step 380 | Training Acc: 0.916 | Test Acc: 0.775 | Test Loss: 0.602 | Test AUC 0.822 (2412.77 s) 
Epoch 69, Step 570 | Training Acc: 0.943 | Test Acc: 0.848 | Test Loss: 0.361 | Test AUC 0.880 (2419.54 s) 
Epoch 69, Step 760 | Training Acc: 0.928 | Test Acc: 0.865 | Test Loss: 0.316 | Test AUC 0.884 (2426.31 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch70.ckpt-70
(*) Entering Epoch 70 (2434.598 s)
Epoch 70, Step 190 | Training Acc: 0.920 | Test Acc: 0.873 | Test Loss: 0.369 | Test AUC 0.892 (2441.34 s) 
Epoch 70, Step 380 | Training Acc: 0.920 | Test Acc: 0.809 | Test Loss: 0.472 | Test AUC 0.843 (2448.09 s) 
Epoch 70, Step 570 | Training Acc: 0.926 | Test Acc: 0.805 | Test Loss: 0.505 | Test AUC 0.841 (2454.84 s) 
Epoch 70, Step 760 | Training Acc: 0.932 | Test Acc: 0.854 | Test Loss: 0.397 | Test AUC 0.878 (2461.58 s) 
(*) Entering Epoch 71 (2468.894 s)
Epoch 71, Step 190 | Training Acc: 0.926 | Test Acc: 0.867 | Test Loss: 0.342 | Test AUC 0.888 (2475.63 s) 
Epoch 71, Step 380 | Training Acc: 0.910 | Test Acc: 0.881 | Test Loss: 0.316 | Test AUC 0.885 (2482.37 s) 
Epoch 71, Step 570 | Training Acc: 0.914 | Test Acc: 0.879 | Test Loss: 0.289 | Test AUC 0.887 (2489.11 s) 
Epoch 71, Step 760 | Training Acc: 0.900 | Test Acc: 0.867 | Test Loss: 0.300 | Test AUC 0.875 (2495.87 s) 
(*) Entering Epoch 72 (2503.165 s)
Epoch 72, Step 190 | Training Acc: 0.914 | Test Acc: 0.891 | Test Loss: 0.281 | Test AUC 0.898 (2509.92 s) 
Epoch 72, Step 380 | Training Acc: 0.887 | Test Acc: 0.822 | Test Loss: 0.452 | Test AUC 0.832 (2516.70 s) 
Epoch 72, Step 570 | Training Acc: 0.939 | Test Acc: 0.873 | Test Loss: 0.341 | Test AUC 0.893 (2523.43 s) 
Epoch 72, Step 760 | Training Acc: 0.922 | Test Acc: 0.779 | Test Loss: 0.547 | Test AUC 0.827 (2530.17 s) 
(*) Entering Epoch 73 (2537.525 s)
Epoch 73, Step 190 | Training Acc: 0.908 | Test Acc: 0.811 | Test Loss: 0.455 | Test AUC 0.838 (2544.27 s) 
Epoch 73, Step 380 | Training Acc: 0.936 | Test Acc: 0.865 | Test Loss: 0.388 | Test AUC 0.893 (2551.08 s) 
Epoch 73, Step 570 | Training Acc: 0.943 | Test Acc: 0.820 | Test Loss: 0.474 | Test AUC 0.858 (2557.85 s) 
Epoch 73, Step 760 | Training Acc: 0.918 | Test Acc: 0.811 | Test Loss: 0.446 | Test AUC 0.844 (2564.61 s) 
(*) Entering Epoch 74 (2571.953 s)
Epoch 74, Step 190 | Training Acc: 0.922 | Test Acc: 0.889 | Test Loss: 0.313 | Test AUC 0.899 (2578.70 s) 
Epoch 74, Step 380 | Training Acc: 0.938 | Test Acc: 0.889 | Test Loss: 0.279 | Test AUC 0.905 (2585.44 s) 
Epoch 74, Step 570 | Training Acc: 0.920 | Test Acc: 0.879 | Test Loss: 0.320 | Test AUC 0.888 (2592.19 s) 
Epoch 74, Step 760 | Training Acc: 0.916 | Test Acc: 0.869 | Test Loss: 0.329 | Test AUC 0.885 (2598.91 s) 
(*) Entering Epoch 75 (2606.265 s)
Epoch 75, Step 190 | Training Acc: 0.941 | Test Acc: 0.891 | Test Loss: 0.325 | Test AUC 0.911 (2613.04 s) 
Epoch 75, Step 380 | Training Acc: 0.922 | Test Acc: 0.867 | Test Loss: 0.334 | Test AUC 0.890 (2619.79 s) 
Epoch 75, Step 570 | Training Acc: 0.918 | Test Acc: 0.871 | Test Loss: 0.337 | Test AUC 0.884 (2626.54 s) 
Epoch 75, Step 760 | Training Acc: 0.934 | Test Acc: 0.883 | Test Loss: 0.272 | Test AUC 0.905 (2633.28 s) 
(*) Entering Epoch 76 (2640.612 s)
Epoch 76, Step 190 | Training Acc: 0.926 | Test Acc: 0.875 | Test Loss: 0.367 | Test AUC 0.894 (2647.35 s) 
Epoch 76, Step 380 | Training Acc: 0.939 | Test Acc: 0.863 | Test Loss: 0.350 | Test AUC 0.891 (2654.08 s) 
Epoch 76, Step 570 | Training Acc: 0.926 | Test Acc: 0.865 | Test Loss: 0.367 | Test AUC 0.886 (2660.82 s) 
Epoch 76, Step 760 | Training Acc: 0.934 | Test Acc: 0.893 | Test Loss: 0.284 | Test AUC 0.907 (2667.57 s) 
(*) Entering Epoch 77 (2674.908 s)
Epoch 77, Step 190 | Training Acc: 0.926 | Test Acc: 0.887 | Test Loss: 0.298 | Test AUC 0.901 (2681.68 s) 
Epoch 77, Step 380 | Training Acc: 0.916 | Test Acc: 0.871 | Test Loss: 0.316 | Test AUC 0.884 (2688.44 s) 
Epoch 77, Step 570 | Training Acc: 0.922 | Test Acc: 0.902 | Test Loss: 0.274 | Test AUC 0.905 (2695.21 s) 
Epoch 77, Step 760 | Training Acc: 0.930 | Test Acc: 0.875 | Test Loss: 0.328 | Test AUC 0.895 (2701.96 s) 
(*) Entering Epoch 78 (2709.293 s)
Epoch 78, Step 190 | Training Acc: 0.912 | Test Acc: 0.900 | Test Loss: 0.251 | Test AUC 0.899 (2716.03 s) 
Epoch 78, Step 380 | Training Acc: 0.938 | Test Acc: 0.838 | Test Loss: 0.417 | Test AUC 0.875 (2722.76 s) 
Epoch 78, Step 570 | Training Acc: 0.922 | Test Acc: 0.852 | Test Loss: 0.355 | Test AUC 0.873 (2729.52 s) 
Epoch 78, Step 760 | Training Acc: 0.939 | Test Acc: 0.779 | Test Loss: 0.545 | Test AUC 0.835 (2736.29 s) 
(*) Entering Epoch 79 (2743.630 s)
Epoch 79, Step 190 | Training Acc: 0.924 | Test Acc: 0.848 | Test Loss: 0.415 | Test AUC 0.873 (2750.40 s) 
Epoch 79, Step 380 | Training Acc: 0.930 | Test Acc: 0.820 | Test Loss: 0.482 | Test AUC 0.861 (2757.14 s) 
Epoch 79, Step 570 | Training Acc: 0.916 | Test Acc: 0.846 | Test Loss: 0.418 | Test AUC 0.866 (2763.88 s) 
Epoch 79, Step 760 | Training Acc: 0.916 | Test Acc: 0.803 | Test Loss: 0.532 | Test AUC 0.832 (2770.63 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch80.ckpt-80
(*) Entering Epoch 80 (2778.948 s)
Epoch 80, Step 190 | Training Acc: 0.951 | Test Acc: 0.871 | Test Loss: 0.402 | Test AUC 0.904 (2785.68 s) 
Epoch 80, Step 380 | Training Acc: 0.938 | Test Acc: 0.863 | Test Loss: 0.318 | Test AUC 0.881 (2792.43 s) 
Epoch 80, Step 570 | Training Acc: 0.920 | Test Acc: 0.879 | Test Loss: 0.310 | Test AUC 0.893 (2799.18 s) 
Epoch 80, Step 760 | Training Acc: 0.924 | Test Acc: 0.889 | Test Loss: 0.283 | Test AUC 0.898 (2805.93 s) 
(*) Entering Epoch 81 (2813.251 s)
Epoch 81, Step 190 | Training Acc: 0.916 | Test Acc: 0.770 | Test Loss: 0.521 | Test AUC 0.815 (2819.98 s) 
Epoch 81, Step 380 | Training Acc: 0.936 | Test Acc: 0.816 | Test Loss: 0.479 | Test AUC 0.858 (2826.72 s) 
Epoch 81, Step 570 | Training Acc: 0.930 | Test Acc: 0.848 | Test Loss: 0.358 | Test AUC 0.877 (2833.49 s) 
Epoch 81, Step 760 | Training Acc: 0.910 | Test Acc: 0.859 | Test Loss: 0.375 | Test AUC 0.877 (2840.22 s) 
(*) Entering Epoch 82 (2847.582 s)
Epoch 82, Step 190 | Training Acc: 0.939 | Test Acc: 0.797 | Test Loss: 0.526 | Test AUC 0.855 (2854.34 s) 
Epoch 82, Step 380 | Training Acc: 0.930 | Test Acc: 0.844 | Test Loss: 0.448 | Test AUC 0.873 (2861.10 s) 
Epoch 82, Step 570 | Training Acc: 0.898 | Test Acc: 0.896 | Test Loss: 0.294 | Test AUC 0.888 (2867.85 s) 
Epoch 82, Step 760 | Training Acc: 0.924 | Test Acc: 0.842 | Test Loss: 0.382 | Test AUC 0.872 (2874.59 s) 
(*) Entering Epoch 83 (2881.920 s)
Epoch 83, Step 190 | Training Acc: 0.924 | Test Acc: 0.857 | Test Loss: 0.391 | Test AUC 0.887 (2888.64 s) 
Epoch 83, Step 380 | Training Acc: 0.947 | Test Acc: 0.898 | Test Loss: 0.275 | Test AUC 0.919 (2895.38 s) 
Epoch 83, Step 570 | Training Acc: 0.920 | Test Acc: 0.879 | Test Loss: 0.333 | Test AUC 0.890 (2902.14 s) 
Epoch 83, Step 760 | Training Acc: 0.926 | Test Acc: 0.883 | Test Loss: 0.341 | Test AUC 0.898 (2908.88 s) 
(*) Entering Epoch 84 (2916.186 s)
Epoch 84, Step 190 | Training Acc: 0.920 | Test Acc: 0.801 | Test Loss: 0.504 | Test AUC 0.843 (2922.91 s) 
Epoch 84, Step 380 | Training Acc: 0.945 | Test Acc: 0.838 | Test Loss: 0.459 | Test AUC 0.873 (2929.65 s) 
Epoch 84, Step 570 | Training Acc: 0.920 | Test Acc: 0.857 | Test Loss: 0.386 | Test AUC 0.879 (2936.38 s) 
Epoch 84, Step 760 | Training Acc: 0.938 | Test Acc: 0.883 | Test Loss: 0.319 | Test AUC 0.904 (2943.12 s) 
(*) Entering Epoch 85 (2950.432 s)
Epoch 85, Step 190 | Training Acc: 0.926 | Test Acc: 0.893 | Test Loss: 0.279 | Test AUC 0.904 (2957.19 s) 
Epoch 85, Step 380 | Training Acc: 0.939 | Test Acc: 0.865 | Test Loss: 0.340 | Test AUC 0.896 (2963.93 s) 
Epoch 85, Step 570 | Training Acc: 0.936 | Test Acc: 0.898 | Test Loss: 0.293 | Test AUC 0.912 (2970.73 s) 
Epoch 85, Step 760 | Training Acc: 0.932 | Test Acc: 0.877 | Test Loss: 0.297 | Test AUC 0.887 (2977.47 s) 
(*) Entering Epoch 86 (2984.771 s)
Epoch 86, Step 190 | Training Acc: 0.932 | Test Acc: 0.777 | Test Loss: 0.613 | Test AUC 0.833 (2991.51 s) 
Epoch 86, Step 380 | Training Acc: 0.934 | Test Acc: 0.830 | Test Loss: 0.466 | Test AUC 0.856 (2998.26 s) 
Epoch 86, Step 570 | Training Acc: 0.939 | Test Acc: 0.844 | Test Loss: 0.397 | Test AUC 0.879 (3005.01 s) 
Epoch 86, Step 760 | Training Acc: 0.914 | Test Acc: 0.824 | Test Loss: 0.386 | Test AUC 0.851 (3011.77 s) 
(*) Entering Epoch 87 (3019.089 s)
Epoch 87, Step 190 | Training Acc: 0.916 | Test Acc: 0.857 | Test Loss: 0.369 | Test AUC 0.880 (3025.86 s) 
Epoch 87, Step 380 | Training Acc: 0.941 | Test Acc: 0.836 | Test Loss: 0.424 | Test AUC 0.874 (3032.61 s) 
Epoch 87, Step 570 | Training Acc: 0.910 | Test Acc: 0.867 | Test Loss: 0.380 | Test AUC 0.879 (3039.34 s) 
Epoch 87, Step 760 | Training Acc: 0.920 | Test Acc: 0.885 | Test Loss: 0.330 | Test AUC 0.897 (3046.10 s) 
(*) Entering Epoch 88 (3053.411 s)
Epoch 88, Step 190 | Training Acc: 0.941 | Test Acc: 0.877 | Test Loss: 0.353 | Test AUC 0.901 (3060.16 s) 
Epoch 88, Step 380 | Training Acc: 0.928 | Test Acc: 0.879 | Test Loss: 0.356 | Test AUC 0.897 (3066.91 s) 
Epoch 88, Step 570 | Training Acc: 0.943 | Test Acc: 0.871 | Test Loss: 0.332 | Test AUC 0.901 (3073.65 s) 
Epoch 88, Step 760 | Training Acc: 0.932 | Test Acc: 0.875 | Test Loss: 0.277 | Test AUC 0.893 (3080.39 s) 
(*) Entering Epoch 89 (3087.730 s)
Epoch 89, Step 190 | Training Acc: 0.914 | Test Acc: 0.793 | Test Loss: 0.561 | Test AUC 0.834 (3094.47 s) 
Epoch 89, Step 380 | Training Acc: 0.934 | Test Acc: 0.848 | Test Loss: 0.395 | Test AUC 0.876 (3101.19 s) 
Epoch 89, Step 570 | Training Acc: 0.916 | Test Acc: 0.852 | Test Loss: 0.445 | Test AUC 0.866 (3107.92 s) 
Epoch 89, Step 760 | Training Acc: 0.941 | Test Acc: 0.863 | Test Loss: 0.346 | Test AUC 0.894 (3114.67 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch90.ckpt-90
(*) Entering Epoch 90 (3122.997 s)
Epoch 90, Step 190 | Training Acc: 0.934 | Test Acc: 0.895 | Test Loss: 0.291 | Test AUC 0.908 (3129.74 s) 
Epoch 90, Step 380 | Training Acc: 0.920 | Test Acc: 0.902 | Test Loss: 0.279 | Test AUC 0.908 (3136.50 s) 
Epoch 90, Step 570 | Training Acc: 0.920 | Test Acc: 0.875 | Test Loss: 0.333 | Test AUC 0.893 (3143.25 s) 
Epoch 90, Step 760 | Training Acc: 0.938 | Test Acc: 0.896 | Test Loss: 0.335 | Test AUC 0.914 (3150.00 s) 
(*) Entering Epoch 91 (3157.314 s)
Epoch 91, Step 190 | Training Acc: 0.936 | Test Acc: 0.889 | Test Loss: 0.314 | Test AUC 0.906 (3164.05 s) 
Epoch 91, Step 380 | Training Acc: 0.928 | Test Acc: 0.893 | Test Loss: 0.317 | Test AUC 0.900 (3170.82 s) 
Epoch 91, Step 570 | Training Acc: 0.932 | Test Acc: 0.801 | Test Loss: 0.529 | Test AUC 0.854 (3177.59 s) 
Epoch 91, Step 760 | Training Acc: 0.920 | Test Acc: 0.850 | Test Loss: 0.381 | Test AUC 0.870 (3184.36 s) 
(*) Entering Epoch 92 (3191.714 s)
Epoch 92, Step 190 | Training Acc: 0.910 | Test Acc: 0.883 | Test Loss: 0.307 | Test AUC 0.892 (3198.47 s) 
Epoch 92, Step 380 | Training Acc: 0.934 | Test Acc: 0.887 | Test Loss: 0.313 | Test AUC 0.906 (3205.21 s) 
Epoch 92, Step 570 | Training Acc: 0.920 | Test Acc: 0.875 | Test Loss: 0.317 | Test AUC 0.886 (3211.96 s) 
Epoch 92, Step 760 | Training Acc: 0.930 | Test Acc: 0.795 | Test Loss: 0.523 | Test AUC 0.840 (3218.71 s) 
(*) Entering Epoch 93 (3226.008 s)
Epoch 93, Step 190 | Training Acc: 0.938 | Test Acc: 0.787 | Test Loss: 0.591 | Test AUC 0.839 (3232.75 s) 
Epoch 93, Step 380 | Training Acc: 0.928 | Test Acc: 0.795 | Test Loss: 0.605 | Test AUC 0.842 (3239.52 s) 
Epoch 93, Step 570 | Training Acc: 0.953 | Test Acc: 0.832 | Test Loss: 0.531 | Test AUC 0.882 (3246.27 s) 
Epoch 93, Step 760 | Training Acc: 0.891 | Test Acc: 0.863 | Test Loss: 0.317 | Test AUC 0.868 (3253.02 s) 
(*) Entering Epoch 94 (3260.367 s)
Epoch 94, Step 190 | Training Acc: 0.938 | Test Acc: 0.881 | Test Loss: 0.332 | Test AUC 0.902 (3267.12 s) 
Epoch 94, Step 380 | Training Acc: 0.957 | Test Acc: 0.867 | Test Loss: 0.387 | Test AUC 0.908 (3273.86 s) 
Epoch 94, Step 570 | Training Acc: 0.939 | Test Acc: 0.859 | Test Loss: 0.364 | Test AUC 0.891 (3280.62 s) 
Epoch 94, Step 760 | Training Acc: 0.945 | Test Acc: 0.863 | Test Loss: 0.358 | Test AUC 0.900 (3287.37 s) 
(*) Entering Epoch 95 (3294.687 s)
Epoch 95, Step 190 | Training Acc: 0.945 | Test Acc: 0.830 | Test Loss: 0.417 | Test AUC 0.864 (3301.46 s) 
Epoch 95, Step 380 | Training Acc: 0.920 | Test Acc: 0.814 | Test Loss: 0.505 | Test AUC 0.850 (3308.20 s) 
Epoch 95, Step 570 | Training Acc: 0.938 | Test Acc: 0.871 | Test Loss: 0.418 | Test AUC 0.892 (3314.95 s) 
Epoch 95, Step 760 | Training Acc: 0.926 | Test Acc: 0.875 | Test Loss: 0.356 | Test AUC 0.894 (3321.72 s) 
(*) Entering Epoch 96 (3329.034 s)
Epoch 96, Step 190 | Training Acc: 0.932 | Test Acc: 0.838 | Test Loss: 0.432 | Test AUC 0.869 (3335.75 s) 
Epoch 96, Step 380 | Training Acc: 0.928 | Test Acc: 0.855 | Test Loss: 0.448 | Test AUC 0.878 (3342.51 s) 
Epoch 96, Step 570 | Training Acc: 0.928 | Test Acc: 0.867 | Test Loss: 0.338 | Test AUC 0.886 (3349.25 s) 
Epoch 96, Step 760 | Training Acc: 0.939 | Test Acc: 0.805 | Test Loss: 0.491 | Test AUC 0.856 (3356.01 s) 
(*) Entering Epoch 97 (3363.366 s)
Epoch 97, Step 190 | Training Acc: 0.961 | Test Acc: 0.838 | Test Loss: 0.381 | Test AUC 0.881 (3370.12 s) 
Epoch 97, Step 380 | Training Acc: 0.934 | Test Acc: 0.859 | Test Loss: 0.382 | Test AUC 0.888 (3376.88 s) 
Epoch 97, Step 570 | Training Acc: 0.938 | Test Acc: 0.877 | Test Loss: 0.304 | Test AUC 0.898 (3383.61 s) 
Epoch 97, Step 760 | Training Acc: 0.945 | Test Acc: 0.852 | Test Loss: 0.367 | Test AUC 0.888 (3390.38 s) 
(*) Entering Epoch 98 (3397.712 s)
Epoch 98, Step 190 | Training Acc: 0.910 | Test Acc: 0.854 | Test Loss: 0.435 | Test AUC 0.873 (3404.46 s) 
Epoch 98, Step 380 | Training Acc: 0.926 | Test Acc: 0.896 | Test Loss: 0.314 | Test AUC 0.903 (3411.21 s) 
Epoch 98, Step 570 | Training Acc: 0.941 | Test Acc: 0.857 | Test Loss: 0.433 | Test AUC 0.889 (3417.96 s) 
Epoch 98, Step 760 | Training Acc: 0.936 | Test Acc: 0.842 | Test Loss: 0.377 | Test AUC 0.883 (3424.72 s) 
(*) Entering Epoch 99 (3432.046 s)
Epoch 99, Step 190 | Training Acc: 0.947 | Test Acc: 0.814 | Test Loss: 0.500 | Test AUC 0.864 (3438.79 s) 
Epoch 99, Step 380 | Training Acc: 0.938 | Test Acc: 0.867 | Test Loss: 0.395 | Test AUC 0.894 (3445.55 s) 
Epoch 99, Step 570 | Training Acc: 0.934 | Test Acc: 0.818 | Test Loss: 0.438 | Test AUC 0.859 (3452.29 s) 
Epoch 99, Step 760 | Training Acc: 0.943 | Test Acc: 0.865 | Test Loss: 0.424 | Test AUC 0.891 (3459.04 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch100.ckpt-100
(*) Entering Epoch 100 (3467.380 s)
Epoch 100, Step 190 | Training Acc: 0.947 | Test Acc: 0.873 | Test Loss: 0.385 | Test AUC 0.904 (3474.12 s) 
Epoch 100, Step 380 | Training Acc: 0.926 | Test Acc: 0.879 | Test Loss: 0.333 | Test AUC 0.899 (3480.90 s) 
Epoch 100, Step 570 | Training Acc: 0.936 | Test Acc: 0.795 | Test Loss: 0.520 | Test AUC 0.843 (3487.65 s) 
Epoch 100, Step 760 | Training Acc: 0.920 | Test Acc: 0.852 | Test Loss: 0.406 | Test AUC 0.871 (3494.39 s) 
(*) Entering Epoch 101 (3501.742 s)
Epoch 101, Step 190 | Training Acc: 0.934 | Test Acc: 0.902 | Test Loss: 0.265 | Test AUC 0.913 (3508.46 s) 
Epoch 101, Step 380 | Training Acc: 0.930 | Test Acc: 0.812 | Test Loss: 0.470 | Test AUC 0.851 (3515.23 s) 
Epoch 101, Step 570 | Training Acc: 0.922 | Test Acc: 0.863 | Test Loss: 0.454 | Test AUC 0.879 (3521.96 s) 
Epoch 101, Step 760 | Training Acc: 0.938 | Test Acc: 0.850 | Test Loss: 0.374 | Test AUC 0.886 (3528.71 s) 
(*) Entering Epoch 102 (3536.077 s)
Epoch 102, Step 190 | Training Acc: 0.932 | Test Acc: 0.768 | Test Loss: 0.616 | Test AUC 0.821 (3542.85 s) 
Epoch 102, Step 380 | Training Acc: 0.924 | Test Acc: 0.834 | Test Loss: 0.439 | Test AUC 0.868 (3549.59 s) 
Epoch 102, Step 570 | Training Acc: 0.934 | Test Acc: 0.873 | Test Loss: 0.321 | Test AUC 0.895 (3556.35 s) 
Epoch 102, Step 760 | Training Acc: 0.938 | Test Acc: 0.834 | Test Loss: 0.456 | Test AUC 0.869 (3563.09 s) 
(*) Entering Epoch 103 (3570.415 s)
Epoch 103, Step 190 | Training Acc: 0.939 | Test Acc: 0.885 | Test Loss: 0.314 | Test AUC 0.908 (3577.16 s) 
Epoch 103, Step 380 | Training Acc: 0.939 | Test Acc: 0.889 | Test Loss: 0.328 | Test AUC 0.910 (3583.91 s) 
Epoch 103, Step 570 | Training Acc: 0.941 | Test Acc: 0.881 | Test Loss: 0.302 | Test AUC 0.901 (3590.69 s) 
Epoch 103, Step 760 | Training Acc: 0.943 | Test Acc: 0.879 | Test Loss: 0.351 | Test AUC 0.902 (3597.48 s) 
(*) Entering Epoch 104 (3604.826 s)
Epoch 104, Step 190 | Training Acc: 0.941 | Test Acc: 0.869 | Test Loss: 0.388 | Test AUC 0.901 (3611.59 s) 
Epoch 104, Step 380 | Training Acc: 0.928 | Test Acc: 0.797 | Test Loss: 0.583 | Test AUC 0.835 (3618.35 s) 
Epoch 104, Step 570 | Training Acc: 0.926 | Test Acc: 0.830 | Test Loss: 0.437 | Test AUC 0.862 (3625.11 s) 
Epoch 104, Step 760 | Training Acc: 0.947 | Test Acc: 0.828 | Test Loss: 0.476 | Test AUC 0.868 (3631.88 s) 
(*) Entering Epoch 105 (3639.205 s)
Epoch 105, Step 190 | Training Acc: 0.922 | Test Acc: 0.855 | Test Loss: 0.395 | Test AUC 0.878 (3645.96 s) 
Epoch 105, Step 380 | Training Acc: 0.928 | Test Acc: 0.855 | Test Loss: 0.378 | Test AUC 0.882 (3652.74 s) 
Epoch 105, Step 570 | Training Acc: 0.906 | Test Acc: 0.883 | Test Loss: 0.374 | Test AUC 0.888 (3659.54 s) 
Epoch 105, Step 760 | Training Acc: 0.924 | Test Acc: 0.836 | Test Loss: 0.479 | Test AUC 0.867 (3666.33 s) 
(*) Entering Epoch 106 (3673.706 s)
Epoch 106, Step 190 | Training Acc: 0.922 | Test Acc: 0.793 | Test Loss: 0.632 | Test AUC 0.836 (3680.49 s) 
Epoch 106, Step 380 | Training Acc: 0.926 | Test Acc: 0.828 | Test Loss: 0.512 | Test AUC 0.866 (3687.31 s) 
Epoch 106, Step 570 | Training Acc: 0.910 | Test Acc: 0.842 | Test Loss: 0.402 | Test AUC 0.857 (3694.04 s) 
Epoch 106, Step 760 | Training Acc: 0.908 | Test Acc: 0.877 | Test Loss: 0.300 | Test AUC 0.887 (3700.78 s) 
(*) Entering Epoch 107 (3708.135 s)
Epoch 107, Step 190 | Training Acc: 0.918 | Test Acc: 0.842 | Test Loss: 0.436 | Test AUC 0.870 (3714.93 s) 
Epoch 107, Step 380 | Training Acc: 0.934 | Test Acc: 0.832 | Test Loss: 0.407 | Test AUC 0.875 (3721.71 s) 
Epoch 107, Step 570 | Training Acc: 0.967 | Test Acc: 0.879 | Test Loss: 0.323 | Test AUC 0.920 (3728.47 s) 
Epoch 107, Step 760 | Training Acc: 0.930 | Test Acc: 0.855 | Test Loss: 0.338 | Test AUC 0.882 (3735.20 s) 
(*) Entering Epoch 108 (3742.529 s)
Epoch 108, Step 190 | Training Acc: 0.957 | Test Acc: 0.896 | Test Loss: 0.323 | Test AUC 0.919 (3749.29 s) 
Epoch 108, Step 380 | Training Acc: 0.949 | Test Acc: 0.807 | Test Loss: 0.536 | Test AUC 0.857 (3756.02 s) 
Epoch 108, Step 570 | Training Acc: 0.943 | Test Acc: 0.811 | Test Loss: 0.461 | Test AUC 0.866 (3762.78 s) 
Epoch 108, Step 760 | Training Acc: 0.943 | Test Acc: 0.852 | Test Loss: 0.370 | Test AUC 0.892 (3769.56 s) 
(*) Entering Epoch 109 (3776.883 s)
Epoch 109, Step 190 | Training Acc: 0.938 | Test Acc: 0.875 | Test Loss: 0.344 | Test AUC 0.898 (3783.62 s) 
Epoch 109, Step 380 | Training Acc: 0.930 | Test Acc: 0.885 | Test Loss: 0.307 | Test AUC 0.901 (3790.36 s) 
Epoch 109, Step 570 | Training Acc: 0.934 | Test Acc: 0.834 | Test Loss: 0.458 | Test AUC 0.866 (3797.10 s) 
Epoch 109, Step 760 | Training Acc: 0.938 | Test Acc: 0.844 | Test Loss: 0.428 | Test AUC 0.880 (3803.84 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch110.ckpt-110
(*) Entering Epoch 110 (3812.146 s)
Epoch 110, Step 190 | Training Acc: 0.945 | Test Acc: 0.863 | Test Loss: 0.412 | Test AUC 0.899 (3818.94 s) 
Epoch 110, Step 380 | Training Acc: 0.936 | Test Acc: 0.877 | Test Loss: 0.365 | Test AUC 0.900 (3825.71 s) 
Epoch 110, Step 570 | Training Acc: 0.934 | Test Acc: 0.887 | Test Loss: 0.296 | Test AUC 0.901 (3832.53 s) 
Epoch 110, Step 760 | Training Acc: 0.939 | Test Acc: 0.779 | Test Loss: 0.577 | Test AUC 0.842 (3839.29 s) 
(*) Entering Epoch 111 (3846.633 s)
Epoch 111, Step 190 | Training Acc: 0.936 | Test Acc: 0.863 | Test Loss: 0.376 | Test AUC 0.893 (3853.39 s) 
Epoch 111, Step 380 | Training Acc: 0.945 | Test Acc: 0.863 | Test Loss: 0.370 | Test AUC 0.900 (3860.13 s) 
Epoch 111, Step 570 | Training Acc: 0.924 | Test Acc: 0.859 | Test Loss: 0.347 | Test AUC 0.883 (3866.87 s) 
Epoch 111, Step 760 | Training Acc: 0.936 | Test Acc: 0.873 | Test Loss: 0.320 | Test AUC 0.894 (3873.62 s) 
(*) Entering Epoch 112 (3880.952 s)
Epoch 112, Step 190 | Training Acc: 0.930 | Test Acc: 0.852 | Test Loss: 0.417 | Test AUC 0.886 (3887.68 s) 
Epoch 112, Step 380 | Training Acc: 0.928 | Test Acc: 0.867 | Test Loss: 0.361 | Test AUC 0.895 (3894.44 s) 
Epoch 112, Step 570 | Training Acc: 0.941 | Test Acc: 0.869 | Test Loss: 0.378 | Test AUC 0.900 (3901.17 s) 
Epoch 112, Step 760 | Training Acc: 0.949 | Test Acc: 0.828 | Test Loss: 0.525 | Test AUC 0.874 (3907.93 s) 
(*) Entering Epoch 113 (3915.256 s)
Epoch 113, Step 190 | Training Acc: 0.939 | Test Acc: 0.859 | Test Loss: 0.394 | Test AUC 0.891 (3922.01 s) 
Epoch 113, Step 380 | Training Acc: 0.941 | Test Acc: 0.900 | Test Loss: 0.296 | Test AUC 0.914 (3928.78 s) 
Epoch 113, Step 570 | Training Acc: 0.949 | Test Acc: 0.859 | Test Loss: 0.369 | Test AUC 0.896 (3935.50 s) 
Epoch 113, Step 760 | Training Acc: 0.898 | Test Acc: 0.883 | Test Loss: 0.357 | Test AUC 0.880 (3942.26 s) 
(*) Entering Epoch 114 (3949.557 s)
Epoch 114, Step 190 | Training Acc: 0.928 | Test Acc: 0.885 | Test Loss: 0.326 | Test AUC 0.899 (3956.31 s) 
Epoch 114, Step 380 | Training Acc: 0.945 | Test Acc: 0.873 | Test Loss: 0.352 | Test AUC 0.906 (3963.04 s) 
Epoch 114, Step 570 | Training Acc: 0.947 | Test Acc: 0.873 | Test Loss: 0.369 | Test AUC 0.904 (3969.83 s) 
Epoch 114, Step 760 | Training Acc: 0.902 | Test Acc: 0.867 | Test Loss: 0.375 | Test AUC 0.879 (3976.59 s) 
(*) Entering Epoch 115 (3983.957 s)
Epoch 115, Step 190 | Training Acc: 0.953 | Test Acc: 0.824 | Test Loss: 0.466 | Test AUC 0.865 (3990.72 s) 
Epoch 115, Step 380 | Training Acc: 0.955 | Test Acc: 0.834 | Test Loss: 0.499 | Test AUC 0.880 (3997.48 s) 
Epoch 115, Step 570 | Training Acc: 0.938 | Test Acc: 0.871 | Test Loss: 0.339 | Test AUC 0.898 (4004.23 s) 
Epoch 115, Step 760 | Training Acc: 0.934 | Test Acc: 0.879 | Test Loss: 0.327 | Test AUC 0.902 (4010.97 s) 
(*) Entering Epoch 116 (4018.312 s)
Epoch 116, Step 190 | Training Acc: 0.922 | Test Acc: 0.898 | Test Loss: 0.306 | Test AUC 0.903 (4025.08 s) 
Epoch 116, Step 380 | Training Acc: 0.953 | Test Acc: 0.861 | Test Loss: 0.401 | Test AUC 0.903 (4031.84 s) 
Epoch 116, Step 570 | Training Acc: 0.945 | Test Acc: 0.891 | Test Loss: 0.309 | Test AUC 0.911 (4038.60 s) 
Epoch 116, Step 760 | Training Acc: 0.928 | Test Acc: 0.863 | Test Loss: 0.454 | Test AUC 0.889 (4045.34 s) 
(*) Entering Epoch 117 (4052.681 s)
Epoch 117, Step 190 | Training Acc: 0.939 | Test Acc: 0.852 | Test Loss: 0.446 | Test AUC 0.888 (4059.43 s) 
Epoch 117, Step 380 | Training Acc: 0.938 | Test Acc: 0.895 | Test Loss: 0.321 | Test AUC 0.914 (4066.24 s) 
Epoch 117, Step 570 | Training Acc: 0.936 | Test Acc: 0.873 | Test Loss: 0.326 | Test AUC 0.900 (4073.03 s) 
Epoch 117, Step 760 | Training Acc: 0.953 | Test Acc: 0.877 | Test Loss: 0.314 | Test AUC 0.907 (4079.76 s) 
(*) Entering Epoch 118 (4087.113 s)
Epoch 118, Step 190 | Training Acc: 0.926 | Test Acc: 0.867 | Test Loss: 0.389 | Test AUC 0.893 (4093.88 s) 
Epoch 118, Step 380 | Training Acc: 0.945 | Test Acc: 0.836 | Test Loss: 0.436 | Test AUC 0.873 (4100.62 s) 
Epoch 118, Step 570 | Training Acc: 0.936 | Test Acc: 0.869 | Test Loss: 0.350 | Test AUC 0.891 (4107.39 s) 
Epoch 118, Step 760 | Training Acc: 0.934 | Test Acc: 0.879 | Test Loss: 0.361 | Test AUC 0.901 (4114.16 s) 
(*) Entering Epoch 119 (4121.486 s)
Epoch 119, Step 190 | Training Acc: 0.965 | Test Acc: 0.881 | Test Loss: 0.330 | Test AUC 0.914 (4128.21 s) 
Epoch 119, Step 380 | Training Acc: 0.943 | Test Acc: 0.873 | Test Loss: 0.384 | Test AUC 0.903 (4134.96 s) 
Epoch 119, Step 570 | Training Acc: 0.941 | Test Acc: 0.826 | Test Loss: 0.468 | Test AUC 0.864 (4141.73 s) 
Epoch 119, Step 760 | Training Acc: 0.943 | Test Acc: 0.887 | Test Loss: 0.320 | Test AUC 0.907 (4148.48 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch120.ckpt-120
(*) Entering Epoch 120 (4156.803 s)
Epoch 120, Step 190 | Training Acc: 0.967 | Test Acc: 0.875 | Test Loss: 0.342 | Test AUC 0.916 (4163.57 s) 
Epoch 120, Step 380 | Training Acc: 0.949 | Test Acc: 0.871 | Test Loss: 0.369 | Test AUC 0.902 (4170.33 s) 
Epoch 120, Step 570 | Training Acc: 0.930 | Test Acc: 0.883 | Test Loss: 0.305 | Test AUC 0.900 (4177.09 s) 
Epoch 120, Step 760 | Training Acc: 0.932 | Test Acc: 0.877 | Test Loss: 0.350 | Test AUC 0.898 (4183.87 s) 
(*) Entering Epoch 121 (4191.195 s)
Epoch 121, Step 190 | Training Acc: 0.936 | Test Acc: 0.867 | Test Loss: 0.349 | Test AUC 0.890 (4197.94 s) 
Epoch 121, Step 380 | Training Acc: 0.926 | Test Acc: 0.873 | Test Loss: 0.317 | Test AUC 0.893 (4204.69 s) 
Epoch 121, Step 570 | Training Acc: 0.947 | Test Acc: 0.906 | Test Loss: 0.255 | Test AUC 0.920 (4211.45 s) 
Epoch 121, Step 760 | Training Acc: 0.930 | Test Acc: 0.883 | Test Loss: 0.319 | Test AUC 0.902 (4218.21 s) 
(*) Entering Epoch 122 (4225.564 s)
Epoch 122, Step 190 | Training Acc: 0.938 | Test Acc: 0.855 | Test Loss: 0.477 | Test AUC 0.884 (4232.31 s) 
Epoch 122, Step 380 | Training Acc: 0.936 | Test Acc: 0.838 | Test Loss: 0.556 | Test AUC 0.876 (4239.10 s) 
Epoch 122, Step 570 | Training Acc: 0.930 | Test Acc: 0.887 | Test Loss: 0.329 | Test AUC 0.898 (4245.85 s) 
Epoch 122, Step 760 | Training Acc: 0.939 | Test Acc: 0.879 | Test Loss: 0.368 | Test AUC 0.902 (4252.59 s) 
(*) Entering Epoch 123 (4259.898 s)
Epoch 123, Step 190 | Training Acc: 0.934 | Test Acc: 0.832 | Test Loss: 0.522 | Test AUC 0.873 (4266.65 s) 
Epoch 123, Step 380 | Training Acc: 0.951 | Test Acc: 0.875 | Test Loss: 0.344 | Test AUC 0.908 (4273.42 s) 
Epoch 123, Step 570 | Training Acc: 0.951 | Test Acc: 0.893 | Test Loss: 0.333 | Test AUC 0.915 (4280.16 s) 
Epoch 123, Step 760 | Training Acc: 0.934 | Test Acc: 0.904 | Test Loss: 0.304 | Test AUC 0.919 (4286.90 s) 
(*) Entering Epoch 124 (4294.219 s)
Epoch 124, Step 190 | Training Acc: 0.926 | Test Acc: 0.824 | Test Loss: 0.486 | Test AUC 0.860 (4300.97 s) 
Epoch 124, Step 380 | Training Acc: 0.941 | Test Acc: 0.834 | Test Loss: 0.462 | Test AUC 0.872 (4307.71 s) 
Epoch 124, Step 570 | Training Acc: 0.906 | Test Acc: 0.828 | Test Loss: 0.432 | Test AUC 0.852 (4314.45 s) 
Epoch 124, Step 760 | Training Acc: 0.932 | Test Acc: 0.828 | Test Loss: 0.478 | Test AUC 0.866 (4321.20 s) 
(*) Entering Epoch 125 (4328.536 s)
Epoch 125, Step 190 | Training Acc: 0.943 | Test Acc: 0.867 | Test Loss: 0.368 | Test AUC 0.897 (4335.32 s) 
Epoch 125, Step 380 | Training Acc: 0.949 | Test Acc: 0.855 | Test Loss: 0.367 | Test AUC 0.897 (4342.07 s) 
Epoch 125, Step 570 | Training Acc: 0.936 | Test Acc: 0.873 | Test Loss: 0.315 | Test AUC 0.893 (4348.85 s) 
Epoch 125, Step 760 | Training Acc: 0.928 | Test Acc: 0.812 | Test Loss: 0.514 | Test AUC 0.853 (4355.62 s) 
(*) Entering Epoch 126 (4362.957 s)
Epoch 126, Step 190 | Training Acc: 0.932 | Test Acc: 0.807 | Test Loss: 0.592 | Test AUC 0.847 (4369.71 s) 
Epoch 126, Step 380 | Training Acc: 0.936 | Test Acc: 0.818 | Test Loss: 0.533 | Test AUC 0.860 (4376.46 s) 
Epoch 126, Step 570 | Training Acc: 0.928 | Test Acc: 0.846 | Test Loss: 0.463 | Test AUC 0.866 (4383.21 s) 
Epoch 126, Step 760 | Training Acc: 0.945 | Test Acc: 0.844 | Test Loss: 0.444 | Test AUC 0.883 (4389.97 s) 
(*) Entering Epoch 127 (4397.296 s)
Epoch 127, Step 190 | Training Acc: 0.953 | Test Acc: 0.891 | Test Loss: 0.330 | Test AUC 0.916 (4404.03 s) 
Epoch 127, Step 380 | Training Acc: 0.934 | Test Acc: 0.879 | Test Loss: 0.300 | Test AUC 0.900 (4410.80 s) 
Epoch 127, Step 570 | Training Acc: 0.939 | Test Acc: 0.887 | Test Loss: 0.338 | Test AUC 0.907 (4417.59 s) 
Epoch 127, Step 760 | Training Acc: 0.943 | Test Acc: 0.875 | Test Loss: 0.346 | Test AUC 0.904 (4424.35 s) 
(*) Entering Epoch 128 (4431.666 s)
Epoch 128, Step 190 | Training Acc: 0.939 | Test Acc: 0.881 | Test Loss: 0.383 | Test AUC 0.902 (4438.41 s) 
Epoch 128, Step 380 | Training Acc: 0.957 | Test Acc: 0.879 | Test Loss: 0.357 | Test AUC 0.915 (4445.17 s) 
Epoch 128, Step 570 | Training Acc: 0.951 | Test Acc: 0.889 | Test Loss: 0.358 | Test AUC 0.914 (4451.94 s) 
Epoch 128, Step 760 | Training Acc: 0.951 | Test Acc: 0.877 | Test Loss: 0.322 | Test AUC 0.906 (4458.73 s) 
(*) Entering Epoch 129 (4466.072 s)
Epoch 129, Step 190 | Training Acc: 0.959 | Test Acc: 0.902 | Test Loss: 0.282 | Test AUC 0.927 (4472.83 s) 
Epoch 129, Step 380 | Training Acc: 0.928 | Test Acc: 0.850 | Test Loss: 0.350 | Test AUC 0.873 (4479.59 s) 
Epoch 129, Step 570 | Training Acc: 0.928 | Test Acc: 0.865 | Test Loss: 0.338 | Test AUC 0.890 (4486.33 s) 
Epoch 129, Step 760 | Training Acc: 0.930 | Test Acc: 0.832 | Test Loss: 0.457 | Test AUC 0.869 (4493.11 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch130.ckpt-130
(*) Entering Epoch 130 (4501.454 s)
Epoch 130, Step 190 | Training Acc: 0.941 | Test Acc: 0.832 | Test Loss: 0.545 | Test AUC 0.878 (4508.20 s) 
Epoch 130, Step 380 | Training Acc: 0.947 | Test Acc: 0.873 | Test Loss: 0.418 | Test AUC 0.903 (4514.96 s) 
Epoch 130, Step 570 | Training Acc: 0.963 | Test Acc: 0.811 | Test Loss: 0.532 | Test AUC 0.869 (4521.72 s) 
Epoch 130, Step 760 | Training Acc: 0.932 | Test Acc: 0.838 | Test Loss: 0.456 | Test AUC 0.876 (4528.47 s) 
(*) Entering Epoch 131 (4535.815 s)
Epoch 131, Step 190 | Training Acc: 0.961 | Test Acc: 0.791 | Test Loss: 0.564 | Test AUC 0.863 (4542.57 s) 
Epoch 131, Step 380 | Training Acc: 0.938 | Test Acc: 0.838 | Test Loss: 0.448 | Test AUC 0.874 (4549.31 s) 
Epoch 131, Step 570 | Training Acc: 0.938 | Test Acc: 0.857 | Test Loss: 0.416 | Test AUC 0.888 (4556.07 s) 
Epoch 131, Step 760 | Training Acc: 0.941 | Test Acc: 0.887 | Test Loss: 0.325 | Test AUC 0.906 (4562.85 s) 
(*) Entering Epoch 132 (4570.193 s)
Epoch 132, Step 190 | Training Acc: 0.930 | Test Acc: 0.852 | Test Loss: 0.418 | Test AUC 0.880 (4576.96 s) 
Epoch 132, Step 380 | Training Acc: 0.938 | Test Acc: 0.863 | Test Loss: 0.369 | Test AUC 0.896 (4583.71 s) 
Epoch 132, Step 570 | Training Acc: 0.936 | Test Acc: 0.867 | Test Loss: 0.388 | Test AUC 0.893 (4590.44 s) 
Epoch 132, Step 760 | Training Acc: 0.949 | Test Acc: 0.893 | Test Loss: 0.301 | Test AUC 0.917 (4597.23 s) 
(*) Entering Epoch 133 (4604.540 s)
Epoch 133, Step 190 | Training Acc: 0.930 | Test Acc: 0.822 | Test Loss: 0.484 | Test AUC 0.854 (4611.31 s) 
Epoch 133, Step 380 | Training Acc: 0.930 | Test Acc: 0.811 | Test Loss: 0.583 | Test AUC 0.860 (4618.04 s) 
Epoch 133, Step 570 | Training Acc: 0.939 | Test Acc: 0.854 | Test Loss: 0.432 | Test AUC 0.886 (4625.02 s) 
Epoch 133, Step 760 | Training Acc: 0.949 | Test Acc: 0.869 | Test Loss: 0.365 | Test AUC 0.899 (4631.78 s) 
(*) Entering Epoch 134 (4639.118 s)
Epoch 134, Step 190 | Training Acc: 0.938 | Test Acc: 0.842 | Test Loss: 0.424 | Test AUC 0.881 (4645.88 s) 
Epoch 134, Step 380 | Training Acc: 0.924 | Test Acc: 0.834 | Test Loss: 0.517 | Test AUC 0.865 (4652.63 s) 
Epoch 134, Step 570 | Training Acc: 0.945 | Test Acc: 0.863 | Test Loss: 0.338 | Test AUC 0.899 (4659.42 s) 
Epoch 134, Step 760 | Training Acc: 0.947 | Test Acc: 0.852 | Test Loss: 0.461 | Test AUC 0.887 (4666.17 s) 
(*) Entering Epoch 135 (4673.476 s)
Epoch 135, Step 190 | Training Acc: 0.947 | Test Acc: 0.859 | Test Loss: 0.402 | Test AUC 0.892 (4680.23 s) 
Epoch 135, Step 380 | Training Acc: 0.939 | Test Acc: 0.883 | Test Loss: 0.338 | Test AUC 0.902 (4686.96 s) 
Epoch 135, Step 570 | Training Acc: 0.951 | Test Acc: 0.869 | Test Loss: 0.415 | Test AUC 0.907 (4693.71 s) 
Epoch 135, Step 760 | Training Acc: 0.934 | Test Acc: 0.867 | Test Loss: 0.386 | Test AUC 0.895 (4700.46 s) 
(*) Entering Epoch 136 (4707.767 s)
Epoch 136, Step 190 | Training Acc: 0.955 | Test Acc: 0.877 | Test Loss: 0.340 | Test AUC 0.912 (4714.54 s) 
Epoch 136, Step 380 | Training Acc: 0.936 | Test Acc: 0.865 | Test Loss: 0.402 | Test AUC 0.891 (4721.27 s) 
Epoch 136, Step 570 | Training Acc: 0.939 | Test Acc: 0.896 | Test Loss: 0.387 | Test AUC 0.913 (4728.05 s) 
Epoch 136, Step 760 | Training Acc: 0.951 | Test Acc: 0.885 | Test Loss: 0.316 | Test AUC 0.916 (4734.79 s) 
(*) Entering Epoch 137 (4742.107 s)
Epoch 137, Step 190 | Training Acc: 0.945 | Test Acc: 0.820 | Test Loss: 0.502 | Test AUC 0.870 (4748.86 s) 
Epoch 137, Step 380 | Training Acc: 0.947 | Test Acc: 0.859 | Test Loss: 0.424 | Test AUC 0.896 (4755.62 s) 
Epoch 137, Step 570 | Training Acc: 0.924 | Test Acc: 0.824 | Test Loss: 0.489 | Test AUC 0.861 (4762.37 s) 
Epoch 137, Step 760 | Training Acc: 0.955 | Test Acc: 0.836 | Test Loss: 0.431 | Test AUC 0.883 (4769.12 s) 
(*) Entering Epoch 138 (4776.431 s)
Epoch 138, Step 190 | Training Acc: 0.936 | Test Acc: 0.854 | Test Loss: 0.392 | Test AUC 0.885 (4783.18 s) 
Epoch 138, Step 380 | Training Acc: 0.945 | Test Acc: 0.865 | Test Loss: 0.372 | Test AUC 0.899 (4789.92 s) 
Epoch 138, Step 570 | Training Acc: 0.936 | Test Acc: 0.838 | Test Loss: 0.433 | Test AUC 0.877 (4796.69 s) 
Epoch 138, Step 760 | Training Acc: 0.930 | Test Acc: 0.846 | Test Loss: 0.431 | Test AUC 0.879 (4803.44 s) 
(*) Entering Epoch 139 (4810.729 s)
Epoch 139, Step 190 | Training Acc: 0.928 | Test Acc: 0.854 | Test Loss: 0.423 | Test AUC 0.879 (4817.47 s) 
Epoch 139, Step 380 | Training Acc: 0.941 | Test Acc: 0.824 | Test Loss: 0.479 | Test AUC 0.873 (4824.23 s) 
Epoch 139, Step 570 | Training Acc: 0.934 | Test Acc: 0.840 | Test Loss: 0.483 | Test AUC 0.872 (4830.99 s) 
Epoch 139, Step 760 | Training Acc: 0.936 | Test Acc: 0.863 | Test Loss: 0.391 | Test AUC 0.888 (4837.74 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch140.ckpt-140
(*) Entering Epoch 140 (4846.012 s)
Epoch 140, Step 190 | Training Acc: 0.939 | Test Acc: 0.869 | Test Loss: 0.397 | Test AUC 0.900 (4852.77 s) 
Epoch 140, Step 380 | Training Acc: 0.934 | Test Acc: 0.896 | Test Loss: 0.289 | Test AUC 0.906 (4859.52 s) 
Epoch 140, Step 570 | Training Acc: 0.938 | Test Acc: 0.875 | Test Loss: 0.347 | Test AUC 0.903 (4866.28 s) 
Epoch 140, Step 760 | Training Acc: 0.943 | Test Acc: 0.877 | Test Loss: 0.370 | Test AUC 0.901 (4873.04 s) 
(*) Entering Epoch 141 (4880.366 s)
Epoch 141, Step 190 | Training Acc: 0.936 | Test Acc: 0.885 | Test Loss: 0.340 | Test AUC 0.903 (4887.13 s) 
Epoch 141, Step 380 | Training Acc: 0.969 | Test Acc: 0.836 | Test Loss: 0.478 | Test AUC 0.889 (4893.90 s) 
Epoch 141, Step 570 | Training Acc: 0.953 | Test Acc: 0.877 | Test Loss: 0.384 | Test AUC 0.906 (4900.65 s) 
Epoch 141, Step 760 | Training Acc: 0.939 | Test Acc: 0.865 | Test Loss: 0.400 | Test AUC 0.888 (4907.41 s) 
(*) Entering Epoch 142 (4914.737 s)
Epoch 142, Step 190 | Training Acc: 0.924 | Test Acc: 0.896 | Test Loss: 0.317 | Test AUC 0.904 (4921.47 s) 
Epoch 142, Step 380 | Training Acc: 0.951 | Test Acc: 0.883 | Test Loss: 0.296 | Test AUC 0.911 (4928.24 s) 
Epoch 142, Step 570 | Training Acc: 0.936 | Test Acc: 0.873 | Test Loss: 0.322 | Test AUC 0.899 (4934.97 s) 
Epoch 142, Step 760 | Training Acc: 0.938 | Test Acc: 0.865 | Test Loss: 0.424 | Test AUC 0.891 (4941.73 s) 
(*) Entering Epoch 143 (4949.062 s)
Epoch 143, Step 190 | Training Acc: 0.918 | Test Acc: 0.877 | Test Loss: 0.373 | Test AUC 0.890 (4955.89 s) 
Epoch 143, Step 380 | Training Acc: 0.947 | Test Acc: 0.895 | Test Loss: 0.345 | Test AUC 0.916 (4962.65 s) 
Epoch 143, Step 570 | Training Acc: 0.945 | Test Acc: 0.850 | Test Loss: 0.403 | Test AUC 0.879 (4969.40 s) 
Epoch 143, Step 760 | Training Acc: 0.961 | Test Acc: 0.865 | Test Loss: 0.355 | Test AUC 0.903 (4976.15 s) 
(*) Entering Epoch 144 (4983.481 s)
Epoch 144, Step 190 | Training Acc: 0.939 | Test Acc: 0.824 | Test Loss: 0.509 | Test AUC 0.865 (4990.22 s) 
Epoch 144, Step 380 | Training Acc: 0.949 | Test Acc: 0.824 | Test Loss: 0.452 | Test AUC 0.872 (4996.96 s) 
Epoch 144, Step 570 | Training Acc: 0.928 | Test Acc: 0.875 | Test Loss: 0.353 | Test AUC 0.890 (5003.73 s) 
Epoch 144, Step 760 | Training Acc: 0.953 | Test Acc: 0.859 | Test Loss: 0.380 | Test AUC 0.894 (5010.47 s) 
(*) Entering Epoch 145 (5017.821 s)
Epoch 145, Step 190 | Training Acc: 0.945 | Test Acc: 0.875 | Test Loss: 0.374 | Test AUC 0.902 (5024.56 s) 
Epoch 145, Step 380 | Training Acc: 0.953 | Test Acc: 0.861 | Test Loss: 0.426 | Test AUC 0.900 (5031.32 s) 
Epoch 145, Step 570 | Training Acc: 0.943 | Test Acc: 0.855 | Test Loss: 0.443 | Test AUC 0.885 (5038.08 s) 
Epoch 145, Step 760 | Training Acc: 0.945 | Test Acc: 0.812 | Test Loss: 0.519 | Test AUC 0.865 (5044.87 s) 
(*) Entering Epoch 146 (5052.189 s)
Epoch 146, Step 190 | Training Acc: 0.961 | Test Acc: 0.881 | Test Loss: 0.320 | Test AUC 0.915 (5058.99 s) 
Epoch 146, Step 380 | Training Acc: 0.967 | Test Acc: 0.861 | Test Loss: 0.414 | Test AUC 0.909 (5065.80 s) 
Epoch 146, Step 570 | Training Acc: 0.959 | Test Acc: 0.850 | Test Loss: 0.425 | Test AUC 0.896 (5072.61 s) 
Epoch 146, Step 760 | Training Acc: 0.922 | Test Acc: 0.906 | Test Loss: 0.281 | Test AUC 0.907 (5079.46 s) 
(*) Entering Epoch 147 (5086.805 s)
Epoch 147, Step 190 | Training Acc: 0.961 | Test Acc: 0.844 | Test Loss: 0.548 | Test AUC 0.886 (5093.63 s) 
Epoch 147, Step 380 | Training Acc: 0.963 | Test Acc: 0.840 | Test Loss: 0.422 | Test AUC 0.889 (5100.46 s) 
Epoch 147, Step 570 | Training Acc: 0.943 | Test Acc: 0.859 | Test Loss: 0.426 | Test AUC 0.891 (5107.23 s) 
Epoch 147, Step 760 | Training Acc: 0.963 | Test Acc: 0.812 | Test Loss: 0.510 | Test AUC 0.875 (5114.03 s) 
(*) Entering Epoch 148 (5121.337 s)
Epoch 148, Step 190 | Training Acc: 0.943 | Test Acc: 0.846 | Test Loss: 0.499 | Test AUC 0.881 (5128.16 s) 
Epoch 148, Step 380 | Training Acc: 0.934 | Test Acc: 0.863 | Test Loss: 0.416 | Test AUC 0.886 (5134.99 s) 
Epoch 148, Step 570 | Training Acc: 0.945 | Test Acc: 0.869 | Test Loss: 0.367 | Test AUC 0.905 (5141.88 s) 
Epoch 148, Step 760 | Training Acc: 0.943 | Test Acc: 0.869 | Test Loss: 0.397 | Test AUC 0.899 (5148.72 s) 
(*) Entering Epoch 149 (5156.063 s)
Epoch 149, Step 190 | Training Acc: 0.961 | Test Acc: 0.873 | Test Loss: 0.427 | Test AUC 0.914 (5162.83 s) 
Epoch 149, Step 380 | Training Acc: 0.934 | Test Acc: 0.893 | Test Loss: 0.342 | Test AUC 0.908 (5169.60 s) 
Epoch 149, Step 570 | Training Acc: 0.943 | Test Acc: 0.857 | Test Loss: 0.373 | Test AUC 0.888 (5176.37 s) 
Epoch 149, Step 760 | Training Acc: 0.951 | Test Acc: 0.850 | Test Loss: 0.457 | Test AUC 0.895 (5183.18 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch150.ckpt-150
(*) Entering Epoch 150 (5191.530 s)
Epoch 150, Step 190 | Training Acc: 0.934 | Test Acc: 0.869 | Test Loss: 0.356 | Test AUC 0.896 (5198.27 s) 
Epoch 150, Step 380 | Training Acc: 0.938 | Test Acc: 0.883 | Test Loss: 0.408 | Test AUC 0.903 (5205.03 s) 
Epoch 150, Step 570 | Training Acc: 0.936 | Test Acc: 0.885 | Test Loss: 0.394 | Test AUC 0.907 (5211.77 s) 
Epoch 150, Step 760 | Training Acc: 0.945 | Test Acc: 0.887 | Test Loss: 0.309 | Test AUC 0.909 (5218.51 s) 
(*) Entering Epoch 151 (5225.826 s)
Epoch 151, Step 190 | Training Acc: 0.951 | Test Acc: 0.893 | Test Loss: 0.331 | Test AUC 0.920 (5232.56 s) 
Epoch 151, Step 380 | Training Acc: 0.955 | Test Acc: 0.881 | Test Loss: 0.394 | Test AUC 0.912 (5239.32 s) 
Epoch 151, Step 570 | Training Acc: 0.957 | Test Acc: 0.857 | Test Loss: 0.397 | Test AUC 0.905 (5246.05 s) 
Epoch 151, Step 760 | Training Acc: 0.947 | Test Acc: 0.830 | Test Loss: 0.441 | Test AUC 0.871 (5252.79 s) 
(*) Entering Epoch 152 (5260.122 s)
Epoch 152, Step 190 | Training Acc: 0.939 | Test Acc: 0.822 | Test Loss: 0.483 | Test AUC 0.871 (5266.87 s) 
Epoch 152, Step 380 | Training Acc: 0.936 | Test Acc: 0.844 | Test Loss: 0.448 | Test AUC 0.880 (5273.63 s) 
Epoch 152, Step 570 | Training Acc: 0.949 | Test Acc: 0.885 | Test Loss: 0.342 | Test AUC 0.909 (5280.40 s) 
Epoch 152, Step 760 | Training Acc: 0.951 | Test Acc: 0.873 | Test Loss: 0.352 | Test AUC 0.908 (5287.17 s) 
(*) Entering Epoch 153 (5294.495 s)
Epoch 153, Step 190 | Training Acc: 0.953 | Test Acc: 0.895 | Test Loss: 0.305 | Test AUC 0.915 (5301.24 s) 
Epoch 153, Step 380 | Training Acc: 0.959 | Test Acc: 0.865 | Test Loss: 0.381 | Test AUC 0.904 (5307.99 s) 
Epoch 153, Step 570 | Training Acc: 0.951 | Test Acc: 0.883 | Test Loss: 0.370 | Test AUC 0.915 (5314.75 s) 
Epoch 153, Step 760 | Training Acc: 0.949 | Test Acc: 0.869 | Test Loss: 0.370 | Test AUC 0.902 (5321.52 s) 
(*) Entering Epoch 154 (5328.819 s)
Epoch 154, Step 190 | Training Acc: 0.957 | Test Acc: 0.869 | Test Loss: 0.348 | Test AUC 0.905 (5335.58 s) 
Epoch 154, Step 380 | Training Acc: 0.953 | Test Acc: 0.857 | Test Loss: 0.413 | Test AUC 0.897 (5342.33 s) 
Epoch 154, Step 570 | Training Acc: 0.945 | Test Acc: 0.871 | Test Loss: 0.370 | Test AUC 0.905 (5349.07 s) 
Epoch 154, Step 760 | Training Acc: 0.934 | Test Acc: 0.814 | Test Loss: 0.509 | Test AUC 0.861 (5355.82 s) 
(*) Entering Epoch 155 (5363.127 s)
Epoch 155, Step 190 | Training Acc: 0.941 | Test Acc: 0.875 | Test Loss: 0.400 | Test AUC 0.898 (5369.90 s) 
Epoch 155, Step 380 | Training Acc: 0.945 | Test Acc: 0.846 | Test Loss: 0.455 | Test AUC 0.888 (5376.64 s) 
Epoch 155, Step 570 | Training Acc: 0.939 | Test Acc: 0.863 | Test Loss: 0.404 | Test AUC 0.892 (5383.40 s) 
Epoch 155, Step 760 | Training Acc: 0.926 | Test Acc: 0.873 | Test Loss: 0.365 | Test AUC 0.890 (5390.13 s) 
(*) Entering Epoch 156 (5397.472 s)
Epoch 156, Step 190 | Training Acc: 0.959 | Test Acc: 0.861 | Test Loss: 0.435 | Test AUC 0.895 (5404.20 s) 
Epoch 156, Step 380 | Training Acc: 0.941 | Test Acc: 0.852 | Test Loss: 0.397 | Test AUC 0.889 (5410.95 s) 
Epoch 156, Step 570 | Training Acc: 0.953 | Test Acc: 0.854 | Test Loss: 0.431 | Test AUC 0.895 (5417.69 s) 
Epoch 156, Step 760 | Training Acc: 0.951 | Test Acc: 0.875 | Test Loss: 0.362 | Test AUC 0.908 (5424.42 s) 
(*) Entering Epoch 157 (5431.718 s)
Epoch 157, Step 190 | Training Acc: 0.943 | Test Acc: 0.865 | Test Loss: 0.346 | Test AUC 0.898 (5438.47 s) 
Epoch 157, Step 380 | Training Acc: 0.943 | Test Acc: 0.877 | Test Loss: 0.366 | Test AUC 0.901 (5445.20 s) 
Epoch 157, Step 570 | Training Acc: 0.953 | Test Acc: 0.848 | Test Loss: 0.463 | Test AUC 0.887 (5451.95 s) 
Epoch 157, Step 760 | Training Acc: 0.924 | Test Acc: 0.865 | Test Loss: 0.415 | Test AUC 0.886 (5458.72 s) 
(*) Entering Epoch 158 (5466.063 s)
Epoch 158, Step 190 | Training Acc: 0.963 | Test Acc: 0.848 | Test Loss: 0.408 | Test AUC 0.899 (5472.89 s) 
Epoch 158, Step 380 | Training Acc: 0.934 | Test Acc: 0.881 | Test Loss: 0.392 | Test AUC 0.900 (5479.67 s) 
Epoch 158, Step 570 | Training Acc: 0.934 | Test Acc: 0.877 | Test Loss: 0.373 | Test AUC 0.898 (5486.43 s) 
Epoch 158, Step 760 | Training Acc: 0.953 | Test Acc: 0.885 | Test Loss: 0.346 | Test AUC 0.910 (5493.20 s) 
(*) Entering Epoch 159 (5500.512 s)
Epoch 159, Step 190 | Training Acc: 0.967 | Test Acc: 0.877 | Test Loss: 0.358 | Test AUC 0.915 (5507.30 s) 
Epoch 159, Step 380 | Training Acc: 0.939 | Test Acc: 0.873 | Test Loss: 0.322 | Test AUC 0.896 (5514.09 s) 
Epoch 159, Step 570 | Training Acc: 0.943 | Test Acc: 0.883 | Test Loss: 0.352 | Test AUC 0.913 (5520.86 s) 
Epoch 159, Step 760 | Training Acc: 0.955 | Test Acc: 0.896 | Test Loss: 0.350 | Test AUC 0.924 (5527.61 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch160.ckpt-160
(*) Entering Epoch 160 (5535.889 s)
Epoch 160, Step 190 | Training Acc: 0.955 | Test Acc: 0.848 | Test Loss: 0.395 | Test AUC 0.890 (5542.65 s) 
Epoch 160, Step 380 | Training Acc: 0.939 | Test Acc: 0.861 | Test Loss: 0.418 | Test AUC 0.892 (5549.43 s) 
Epoch 160, Step 570 | Training Acc: 0.939 | Test Acc: 0.863 | Test Loss: 0.372 | Test AUC 0.900 (5556.20 s) 
Epoch 160, Step 760 | Training Acc: 0.957 | Test Acc: 0.836 | Test Loss: 0.482 | Test AUC 0.878 (5562.98 s) 
(*) Entering Epoch 161 (5570.294 s)
Epoch 161, Step 190 | Training Acc: 0.936 | Test Acc: 0.854 | Test Loss: 0.468 | Test AUC 0.880 (5577.04 s) 
Epoch 161, Step 380 | Training Acc: 0.945 | Test Acc: 0.840 | Test Loss: 0.507 | Test AUC 0.871 (5583.86 s) 
Epoch 161, Step 570 | Training Acc: 0.932 | Test Acc: 0.814 | Test Loss: 0.523 | Test AUC 0.857 (5590.62 s) 
Epoch 161, Step 760 | Training Acc: 0.953 | Test Acc: 0.865 | Test Loss: 0.380 | Test AUC 0.901 (5597.39 s) 
(*) Entering Epoch 162 (5604.716 s)
Epoch 162, Step 190 | Training Acc: 0.953 | Test Acc: 0.891 | Test Loss: 0.322 | Test AUC 0.916 (5611.49 s) 
Epoch 162, Step 380 | Training Acc: 0.959 | Test Acc: 0.854 | Test Loss: 0.426 | Test AUC 0.897 (5618.23 s) 
Epoch 162, Step 570 | Training Acc: 0.961 | Test Acc: 0.867 | Test Loss: 0.388 | Test AUC 0.910 (5625.00 s) 
Epoch 162, Step 760 | Training Acc: 0.947 | Test Acc: 0.836 | Test Loss: 0.481 | Test AUC 0.878 (5631.75 s) 
(*) Entering Epoch 163 (5639.082 s)
Epoch 163, Step 190 | Training Acc: 0.951 | Test Acc: 0.883 | Test Loss: 0.421 | Test AUC 0.909 (5645.87 s) 
Epoch 163, Step 380 | Training Acc: 0.957 | Test Acc: 0.873 | Test Loss: 0.376 | Test AUC 0.909 (5652.63 s) 
Epoch 163, Step 570 | Training Acc: 0.957 | Test Acc: 0.844 | Test Loss: 0.511 | Test AUC 0.888 (5659.42 s) 
Epoch 163, Step 760 | Training Acc: 0.959 | Test Acc: 0.834 | Test Loss: 0.472 | Test AUC 0.887 (5666.16 s) 
(*) Entering Epoch 164 (5673.491 s)
Epoch 164, Step 190 | Training Acc: 0.951 | Test Acc: 0.871 | Test Loss: 0.429 | Test AUC 0.903 (5680.24 s) 
Epoch 164, Step 380 | Training Acc: 0.951 | Test Acc: 0.818 | Test Loss: 0.576 | Test AUC 0.867 (5687.04 s) 
Epoch 164, Step 570 | Training Acc: 0.949 | Test Acc: 0.811 | Test Loss: 0.501 | Test AUC 0.861 (5693.85 s) 
Epoch 164, Step 760 | Training Acc: 0.945 | Test Acc: 0.867 | Test Loss: 0.392 | Test AUC 0.887 (5700.61 s) 
(*) Entering Epoch 165 (5707.925 s)
Epoch 165, Step 190 | Training Acc: 0.949 | Test Acc: 0.898 | Test Loss: 0.336 | Test AUC 0.919 (5714.67 s) 
Epoch 165, Step 380 | Training Acc: 0.951 | Test Acc: 0.861 | Test Loss: 0.400 | Test AUC 0.902 (5721.42 s) 
Epoch 165, Step 570 | Training Acc: 0.951 | Test Acc: 0.889 | Test Loss: 0.361 | Test AUC 0.914 (5728.16 s) 
Epoch 165, Step 760 | Training Acc: 0.938 | Test Acc: 0.879 | Test Loss: 0.322 | Test AUC 0.904 (5734.93 s) 
(*) Entering Epoch 166 (5742.284 s)
Epoch 166, Step 190 | Training Acc: 0.916 | Test Acc: 0.865 | Test Loss: 0.385 | Test AUC 0.882 (5749.06 s) 
Epoch 166, Step 380 | Training Acc: 0.957 | Test Acc: 0.857 | Test Loss: 0.394 | Test AUC 0.902 (5755.83 s) 
Epoch 166, Step 570 | Training Acc: 0.949 | Test Acc: 0.811 | Test Loss: 0.502 | Test AUC 0.867 (5762.59 s) 
Epoch 166, Step 760 | Training Acc: 0.955 | Test Acc: 0.826 | Test Loss: 0.568 | Test AUC 0.875 (5769.36 s) 
(*) Entering Epoch 167 (5776.679 s)
Epoch 167, Step 190 | Training Acc: 0.963 | Test Acc: 0.857 | Test Loss: 0.371 | Test AUC 0.901 (5783.43 s) 
Epoch 167, Step 380 | Training Acc: 0.947 | Test Acc: 0.850 | Test Loss: 0.438 | Test AUC 0.894 (5790.20 s) 
Epoch 167, Step 570 | Training Acc: 0.951 | Test Acc: 0.916 | Test Loss: 0.273 | Test AUC 0.928 (5796.97 s) 
Epoch 167, Step 760 | Training Acc: 0.947 | Test Acc: 0.900 | Test Loss: 0.352 | Test AUC 0.915 (5803.72 s) 
(*) Entering Epoch 168 (5811.033 s)
Epoch 168, Step 190 | Training Acc: 0.951 | Test Acc: 0.887 | Test Loss: 0.382 | Test AUC 0.913 (5817.79 s) 
Epoch 168, Step 380 | Training Acc: 0.955 | Test Acc: 0.869 | Test Loss: 0.364 | Test AUC 0.910 (5824.61 s) 
Epoch 168, Step 570 | Training Acc: 0.936 | Test Acc: 0.867 | Test Loss: 0.412 | Test AUC 0.893 (5831.38 s) 
Epoch 168, Step 760 | Training Acc: 0.955 | Test Acc: 0.885 | Test Loss: 0.322 | Test AUC 0.918 (5838.15 s) 
(*) Entering Epoch 169 (5845.492 s)
Epoch 169, Step 190 | Training Acc: 0.951 | Test Acc: 0.830 | Test Loss: 0.482 | Test AUC 0.880 (5852.28 s) 
Epoch 169, Step 380 | Training Acc: 0.953 | Test Acc: 0.867 | Test Loss: 0.398 | Test AUC 0.902 (5859.06 s) 
Epoch 169, Step 570 | Training Acc: 0.955 | Test Acc: 0.812 | Test Loss: 0.618 | Test AUC 0.864 (5865.83 s) 
Epoch 169, Step 760 | Training Acc: 0.957 | Test Acc: 0.826 | Test Loss: 0.491 | Test AUC 0.872 (5872.58 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch170.ckpt-170
(*) Entering Epoch 170 (5880.872 s)
Epoch 170, Step 190 | Training Acc: 0.949 | Test Acc: 0.811 | Test Loss: 0.605 | Test AUC 0.865 (5887.64 s) 
Epoch 170, Step 380 | Training Acc: 0.955 | Test Acc: 0.844 | Test Loss: 0.493 | Test AUC 0.889 (5894.40 s) 
Epoch 170, Step 570 | Training Acc: 0.936 | Test Acc: 0.854 | Test Loss: 0.428 | Test AUC 0.888 (5901.18 s) 
Epoch 170, Step 760 | Training Acc: 0.947 | Test Acc: 0.875 | Test Loss: 0.359 | Test AUC 0.901 (5907.93 s) 
(*) Entering Epoch 171 (5915.243 s)
Epoch 171, Step 190 | Training Acc: 0.943 | Test Acc: 0.855 | Test Loss: 0.400 | Test AUC 0.893 (5922.02 s) 
Epoch 171, Step 380 | Training Acc: 0.955 | Test Acc: 0.875 | Test Loss: 0.353 | Test AUC 0.912 (5928.87 s) 
Epoch 171, Step 570 | Training Acc: 0.949 | Test Acc: 0.848 | Test Loss: 0.443 | Test AUC 0.895 (5935.62 s) 
Epoch 171, Step 760 | Training Acc: 0.953 | Test Acc: 0.893 | Test Loss: 0.333 | Test AUC 0.921 (5942.40 s) 
(*) Entering Epoch 172 (5949.712 s)
Epoch 172, Step 190 | Training Acc: 0.965 | Test Acc: 0.869 | Test Loss: 0.422 | Test AUC 0.915 (5956.48 s) 
Epoch 172, Step 380 | Training Acc: 0.934 | Test Acc: 0.859 | Test Loss: 0.359 | Test AUC 0.891 (5963.24 s) 
Epoch 172, Step 570 | Training Acc: 0.934 | Test Acc: 0.852 | Test Loss: 0.414 | Test AUC 0.887 (5970.01 s) 
Epoch 172, Step 760 | Training Acc: 0.959 | Test Acc: 0.840 | Test Loss: 0.493 | Test AUC 0.887 (5976.77 s) 
(*) Entering Epoch 173 (5984.087 s)
Epoch 173, Step 190 | Training Acc: 0.961 | Test Acc: 0.842 | Test Loss: 0.486 | Test AUC 0.884 (5990.85 s) 
Epoch 173, Step 380 | Training Acc: 0.953 | Test Acc: 0.830 | Test Loss: 0.495 | Test AUC 0.880 (5997.62 s) 
Epoch 173, Step 570 | Training Acc: 0.953 | Test Acc: 0.885 | Test Loss: 0.348 | Test AUC 0.907 (6004.40 s) 
Epoch 173, Step 760 | Training Acc: 0.922 | Test Acc: 0.834 | Test Loss: 0.493 | Test AUC 0.863 (6011.13 s) 
(*) Entering Epoch 174 (6018.467 s)
Epoch 174, Step 190 | Training Acc: 0.941 | Test Acc: 0.898 | Test Loss: 0.339 | Test AUC 0.917 (6025.22 s) 
Epoch 174, Step 380 | Training Acc: 0.957 | Test Acc: 0.879 | Test Loss: 0.404 | Test AUC 0.915 (6032.13 s) 
Epoch 174, Step 570 | Training Acc: 0.961 | Test Acc: 0.857 | Test Loss: 0.452 | Test AUC 0.902 (6038.90 s) 
Epoch 174, Step 760 | Training Acc: 0.957 | Test Acc: 0.865 | Test Loss: 0.354 | Test AUC 0.906 (6045.67 s) 
(*) Entering Epoch 175 (6052.971 s)
Epoch 175, Step 190 | Training Acc: 0.951 | Test Acc: 0.842 | Test Loss: 0.434 | Test AUC 0.882 (6059.74 s) 
Epoch 175, Step 380 | Training Acc: 0.955 | Test Acc: 0.859 | Test Loss: 0.429 | Test AUC 0.906 (6066.48 s) 
Epoch 175, Step 570 | Training Acc: 0.934 | Test Acc: 0.838 | Test Loss: 0.489 | Test AUC 0.874 (6073.24 s) 
Epoch 175, Step 760 | Training Acc: 0.953 | Test Acc: 0.920 | Test Loss: 0.291 | Test AUC 0.932 (6081.02 s) [*]
(*) Entering Epoch 176 (6088.348 s)
Epoch 176, Step 190 | Training Acc: 0.945 | Test Acc: 0.871 | Test Loss: 0.368 | Test AUC 0.905 (6095.12 s) 
Epoch 176, Step 380 | Training Acc: 0.959 | Test Acc: 0.865 | Test Loss: 0.401 | Test AUC 0.906 (6101.88 s) 
Epoch 176, Step 570 | Training Acc: 0.955 | Test Acc: 0.844 | Test Loss: 0.460 | Test AUC 0.889 (6108.72 s) 
Epoch 176, Step 760 | Training Acc: 0.945 | Test Acc: 0.836 | Test Loss: 0.491 | Test AUC 0.881 (6115.51 s) 
(*) Entering Epoch 177 (6122.838 s)
Epoch 177, Step 190 | Training Acc: 0.953 | Test Acc: 0.875 | Test Loss: 0.438 | Test AUC 0.910 (6129.61 s) 
Epoch 177, Step 380 | Training Acc: 0.934 | Test Acc: 0.893 | Test Loss: 0.298 | Test AUC 0.911 (6136.40 s) 
Epoch 177, Step 570 | Training Acc: 0.959 | Test Acc: 0.865 | Test Loss: 0.394 | Test AUC 0.907 (6143.18 s) 
Epoch 177, Step 760 | Training Acc: 0.953 | Test Acc: 0.852 | Test Loss: 0.389 | Test AUC 0.892 (6149.93 s) 
(*) Entering Epoch 178 (6157.240 s)
Epoch 178, Step 190 | Training Acc: 0.951 | Test Acc: 0.902 | Test Loss: 0.301 | Test AUC 0.922 (6163.99 s) 
Epoch 178, Step 380 | Training Acc: 0.912 | Test Acc: 0.889 | Test Loss: 0.317 | Test AUC 0.894 (6170.76 s) 
Epoch 178, Step 570 | Training Acc: 0.967 | Test Acc: 0.809 | Test Loss: 0.442 | Test AUC 0.867 (6177.53 s) 
Epoch 178, Step 760 | Training Acc: 0.949 | Test Acc: 0.832 | Test Loss: 0.506 | Test AUC 0.874 (6184.28 s) 
(*) Entering Epoch 179 (6191.607 s)
Epoch 179, Step 190 | Training Acc: 0.959 | Test Acc: 0.875 | Test Loss: 0.364 | Test AUC 0.912 (6198.37 s) 
Epoch 179, Step 380 | Training Acc: 0.961 | Test Acc: 0.867 | Test Loss: 0.414 | Test AUC 0.905 (6205.14 s) 
Epoch 179, Step 570 | Training Acc: 0.943 | Test Acc: 0.885 | Test Loss: 0.286 | Test AUC 0.907 (6211.90 s) 
Epoch 179, Step 760 | Training Acc: 0.932 | Test Acc: 0.877 | Test Loss: 0.392 | Test AUC 0.893 (6218.64 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch180.ckpt-180
(*) Entering Epoch 180 (6226.943 s)
Epoch 180, Step 190 | Training Acc: 0.961 | Test Acc: 0.885 | Test Loss: 0.367 | Test AUC 0.914 (6233.70 s) 
Epoch 180, Step 380 | Training Acc: 0.949 | Test Acc: 0.875 | Test Loss: 0.326 | Test AUC 0.904 (6240.44 s) 
Epoch 180, Step 570 | Training Acc: 0.947 | Test Acc: 0.834 | Test Loss: 0.468 | Test AUC 0.875 (6247.19 s) 
Epoch 180, Step 760 | Training Acc: 0.949 | Test Acc: 0.826 | Test Loss: 0.458 | Test AUC 0.875 (6253.95 s) 
(*) Entering Epoch 181 (6261.270 s)
Epoch 181, Step 190 | Training Acc: 0.939 | Test Acc: 0.879 | Test Loss: 0.345 | Test AUC 0.902 (6268.01 s) 
Epoch 181, Step 380 | Training Acc: 0.949 | Test Acc: 0.867 | Test Loss: 0.428 | Test AUC 0.904 (6274.76 s) 
Epoch 181, Step 570 | Training Acc: 0.957 | Test Acc: 0.848 | Test Loss: 0.428 | Test AUC 0.894 (6281.51 s) 
Epoch 181, Step 760 | Training Acc: 0.957 | Test Acc: 0.828 | Test Loss: 0.579 | Test AUC 0.876 (6288.27 s) 
(*) Entering Epoch 182 (6295.605 s)
Epoch 182, Step 190 | Training Acc: 0.959 | Test Acc: 0.873 | Test Loss: 0.385 | Test AUC 0.905 (6302.36 s) 
Epoch 182, Step 380 | Training Acc: 0.965 | Test Acc: 0.869 | Test Loss: 0.420 | Test AUC 0.912 (6309.13 s) 
Epoch 182, Step 570 | Training Acc: 0.936 | Test Acc: 0.883 | Test Loss: 0.345 | Test AUC 0.903 (6315.87 s) 
Epoch 182, Step 760 | Training Acc: 0.949 | Test Acc: 0.846 | Test Loss: 0.454 | Test AUC 0.885 (6322.61 s) 
(*) Entering Epoch 183 (6329.918 s)
Epoch 183, Step 190 | Training Acc: 0.957 | Test Acc: 0.818 | Test Loss: 0.497 | Test AUC 0.875 (6336.67 s) 
Epoch 183, Step 380 | Training Acc: 0.945 | Test Acc: 0.846 | Test Loss: 0.482 | Test AUC 0.882 (6343.41 s) 
Epoch 183, Step 570 | Training Acc: 0.949 | Test Acc: 0.822 | Test Loss: 0.471 | Test AUC 0.875 (6350.20 s) 
Epoch 183, Step 760 | Training Acc: 0.945 | Test Acc: 0.826 | Test Loss: 0.522 | Test AUC 0.877 (6356.95 s) 
(*) Entering Epoch 184 (6364.273 s)
Epoch 184, Step 190 | Training Acc: 0.961 | Test Acc: 0.881 | Test Loss: 0.407 | Test AUC 0.915 (6371.03 s) 
Epoch 184, Step 380 | Training Acc: 0.947 | Test Acc: 0.805 | Test Loss: 0.557 | Test AUC 0.863 (6377.79 s) 
Epoch 184, Step 570 | Training Acc: 0.947 | Test Acc: 0.828 | Test Loss: 0.441 | Test AUC 0.881 (6384.57 s) 
Epoch 184, Step 760 | Training Acc: 0.959 | Test Acc: 0.875 | Test Loss: 0.336 | Test AUC 0.909 (6391.32 s) 
(*) Entering Epoch 185 (6398.650 s)
Epoch 185, Step 190 | Training Acc: 0.949 | Test Acc: 0.875 | Test Loss: 0.400 | Test AUC 0.905 (6405.40 s) 
Epoch 185, Step 380 | Training Acc: 0.939 | Test Acc: 0.812 | Test Loss: 0.578 | Test AUC 0.856 (6412.17 s) 
Epoch 185, Step 570 | Training Acc: 0.949 | Test Acc: 0.844 | Test Loss: 0.409 | Test AUC 0.886 (6418.94 s) 
Epoch 185, Step 760 | Training Acc: 0.945 | Test Acc: 0.809 | Test Loss: 0.542 | Test AUC 0.869 (6425.71 s) 
(*) Entering Epoch 186 (6433.060 s)
Epoch 186, Step 190 | Training Acc: 0.965 | Test Acc: 0.855 | Test Loss: 0.414 | Test AUC 0.904 (6439.86 s) 
Epoch 186, Step 380 | Training Acc: 0.941 | Test Acc: 0.861 | Test Loss: 0.373 | Test AUC 0.889 (6446.63 s) 
Epoch 186, Step 570 | Training Acc: 0.953 | Test Acc: 0.871 | Test Loss: 0.409 | Test AUC 0.906 (6453.40 s) 
Epoch 186, Step 760 | Training Acc: 0.947 | Test Acc: 0.857 | Test Loss: 0.454 | Test AUC 0.893 (6460.15 s) 
(*) Entering Epoch 187 (6467.491 s)
Epoch 187, Step 190 | Training Acc: 0.955 | Test Acc: 0.861 | Test Loss: 0.419 | Test AUC 0.899 (6474.26 s) 
Epoch 187, Step 380 | Training Acc: 0.941 | Test Acc: 0.832 | Test Loss: 0.516 | Test AUC 0.875 (6481.01 s) 
Epoch 187, Step 570 | Training Acc: 0.947 | Test Acc: 0.883 | Test Loss: 0.369 | Test AUC 0.906 (6487.77 s) 
Epoch 187, Step 760 | Training Acc: 0.961 | Test Acc: 0.881 | Test Loss: 0.383 | Test AUC 0.914 (6494.51 s) 
(*) Entering Epoch 188 (6501.854 s)
Epoch 188, Step 190 | Training Acc: 0.963 | Test Acc: 0.879 | Test Loss: 0.403 | Test AUC 0.915 (6508.61 s) 
Epoch 188, Step 380 | Training Acc: 0.945 | Test Acc: 0.822 | Test Loss: 0.582 | Test AUC 0.872 (6515.40 s) 
Epoch 188, Step 570 | Training Acc: 0.934 | Test Acc: 0.816 | Test Loss: 0.543 | Test AUC 0.863 (6522.18 s) 
Epoch 188, Step 760 | Training Acc: 0.916 | Test Acc: 0.871 | Test Loss: 0.426 | Test AUC 0.888 (6528.93 s) 
(*) Entering Epoch 189 (6536.267 s)
Epoch 189, Step 190 | Training Acc: 0.941 | Test Acc: 0.859 | Test Loss: 0.394 | Test AUC 0.888 (6543.02 s) 
Epoch 189, Step 380 | Training Acc: 0.945 | Test Acc: 0.826 | Test Loss: 0.548 | Test AUC 0.869 (6549.76 s) 
Epoch 189, Step 570 | Training Acc: 0.955 | Test Acc: 0.852 | Test Loss: 0.461 | Test AUC 0.893 (6556.52 s) 
Epoch 189, Step 760 | Training Acc: 0.965 | Test Acc: 0.859 | Test Loss: 0.403 | Test AUC 0.906 (6563.26 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch190.ckpt-190
(*) Entering Epoch 190 (6571.559 s)
Epoch 190, Step 190 | Training Acc: 0.965 | Test Acc: 0.838 | Test Loss: 0.526 | Test AUC 0.891 (6578.31 s) 
Epoch 190, Step 380 | Training Acc: 0.955 | Test Acc: 0.863 | Test Loss: 0.374 | Test AUC 0.900 (6585.06 s) 
Epoch 190, Step 570 | Training Acc: 0.955 | Test Acc: 0.875 | Test Loss: 0.373 | Test AUC 0.908 (6591.80 s) 
Epoch 190, Step 760 | Training Acc: 0.953 | Test Acc: 0.895 | Test Loss: 0.329 | Test AUC 0.913 (6598.54 s) 
(*) Entering Epoch 191 (6605.891 s)
Epoch 191, Step 190 | Training Acc: 0.965 | Test Acc: 0.893 | Test Loss: 0.350 | Test AUC 0.928 (6612.66 s) 
Epoch 191, Step 380 | Training Acc: 0.938 | Test Acc: 0.873 | Test Loss: 0.332 | Test AUC 0.901 (6619.42 s) 
Epoch 191, Step 570 | Training Acc: 0.959 | Test Acc: 0.904 | Test Loss: 0.286 | Test AUC 0.929 (6626.18 s) 
Epoch 191, Step 760 | Training Acc: 0.963 | Test Acc: 0.848 | Test Loss: 0.489 | Test AUC 0.898 (6632.92 s) 
(*) Entering Epoch 192 (6640.228 s)
Epoch 192, Step 190 | Training Acc: 0.938 | Test Acc: 0.912 | Test Loss: 0.343 | Test AUC 0.919 (6646.99 s) 
Epoch 192, Step 380 | Training Acc: 0.951 | Test Acc: 0.877 | Test Loss: 0.416 | Test AUC 0.908 (6653.78 s) 
Epoch 192, Step 570 | Training Acc: 0.957 | Test Acc: 0.883 | Test Loss: 0.361 | Test AUC 0.915 (6660.55 s) 
Epoch 192, Step 760 | Training Acc: 0.926 | Test Acc: 0.891 | Test Loss: 0.360 | Test AUC 0.902 (6667.29 s) 
(*) Entering Epoch 193 (6674.612 s)
Epoch 193, Step 190 | Training Acc: 0.961 | Test Acc: 0.859 | Test Loss: 0.468 | Test AUC 0.902 (6681.39 s) 
Epoch 193, Step 380 | Training Acc: 0.943 | Test Acc: 0.879 | Test Loss: 0.394 | Test AUC 0.901 (6688.17 s) 
Epoch 193, Step 570 | Training Acc: 0.953 | Test Acc: 0.861 | Test Loss: 0.368 | Test AUC 0.906 (6694.93 s) 
Epoch 193, Step 760 | Training Acc: 0.959 | Test Acc: 0.818 | Test Loss: 0.495 | Test AUC 0.870 (6701.68 s) 
(*) Entering Epoch 194 (6708.988 s)
Epoch 194, Step 190 | Training Acc: 0.959 | Test Acc: 0.846 | Test Loss: 0.478 | Test AUC 0.895 (6715.75 s) 
Epoch 194, Step 380 | Training Acc: 0.959 | Test Acc: 0.865 | Test Loss: 0.383 | Test AUC 0.908 (6722.50 s) 
Epoch 194, Step 570 | Training Acc: 0.953 | Test Acc: 0.863 | Test Loss: 0.408 | Test AUC 0.902 (6729.26 s) 
Epoch 194, Step 760 | Training Acc: 0.947 | Test Acc: 0.885 | Test Loss: 0.330 | Test AUC 0.908 (6736.03 s) 
(*) Entering Epoch 195 (6743.369 s)
Epoch 195, Step 190 | Training Acc: 0.939 | Test Acc: 0.863 | Test Loss: 0.374 | Test AUC 0.898 (6750.13 s) 
Epoch 195, Step 380 | Training Acc: 0.951 | Test Acc: 0.863 | Test Loss: 0.394 | Test AUC 0.899 (6756.91 s) 
Epoch 195, Step 570 | Training Acc: 0.953 | Test Acc: 0.863 | Test Loss: 0.396 | Test AUC 0.905 (6763.66 s) 
Epoch 195, Step 760 | Training Acc: 0.953 | Test Acc: 0.895 | Test Loss: 0.327 | Test AUC 0.917 (6770.46 s) 
(*) Entering Epoch 196 (6777.807 s)
Epoch 196, Step 190 | Training Acc: 0.961 | Test Acc: 0.875 | Test Loss: 0.368 | Test AUC 0.913 (6784.56 s) 
Epoch 196, Step 380 | Training Acc: 0.951 | Test Acc: 0.926 | Test Loss: 0.202 | Test AUC 0.932 (6792.25 s) [*]
Epoch 196, Step 570 | Training Acc: 0.943 | Test Acc: 0.861 | Test Loss: 0.411 | Test AUC 0.896 (6799.02 s) 
Epoch 196, Step 760 | Training Acc: 0.953 | Test Acc: 0.887 | Test Loss: 0.374 | Test AUC 0.915 (6805.78 s) 
(*) Entering Epoch 197 (6813.101 s)
Epoch 197, Step 190 | Training Acc: 0.947 | Test Acc: 0.855 | Test Loss: 0.519 | Test AUC 0.885 (6819.87 s) 
Epoch 197, Step 380 | Training Acc: 0.949 | Test Acc: 0.840 | Test Loss: 0.526 | Test AUC 0.880 (6826.64 s) 
Epoch 197, Step 570 | Training Acc: 0.957 | Test Acc: 0.854 | Test Loss: 0.472 | Test AUC 0.898 (6833.40 s) 
Epoch 197, Step 760 | Training Acc: 0.939 | Test Acc: 0.877 | Test Loss: 0.345 | Test AUC 0.899 (6840.18 s) 
(*) Entering Epoch 198 (6847.544 s)
Epoch 198, Step 190 | Training Acc: 0.951 | Test Acc: 0.859 | Test Loss: 0.380 | Test AUC 0.897 (6854.30 s) 
Epoch 198, Step 380 | Training Acc: 0.955 | Test Acc: 0.885 | Test Loss: 0.430 | Test AUC 0.914 (6861.05 s) 
Epoch 198, Step 570 | Training Acc: 0.943 | Test Acc: 0.867 | Test Loss: 0.396 | Test AUC 0.898 (6867.80 s) 
Epoch 198, Step 760 | Training Acc: 0.959 | Test Acc: 0.865 | Test Loss: 0.412 | Test AUC 0.907 (6874.57 s) 
(*) Entering Epoch 199 (6881.914 s)
Epoch 199, Step 190 | Training Acc: 0.932 | Test Acc: 0.918 | Test Loss: 0.282 | Test AUC 0.920 (6888.67 s) 
Epoch 199, Step 380 | Training Acc: 0.949 | Test Acc: 0.906 | Test Loss: 0.327 | Test AUC 0.922 (6895.43 s) 
Epoch 199, Step 570 | Training Acc: 0.955 | Test Acc: 0.863 | Test Loss: 0.457 | Test AUC 0.899 (6902.19 s) 
Epoch 199, Step 760 | Training Acc: 0.945 | Test Acc: 0.877 | Test Loss: 0.388 | Test AUC 0.907 (6908.97 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch200.ckpt-200
(*) Entering Epoch 200 (6917.454 s)
Epoch 200, Step 190 | Training Acc: 0.949 | Test Acc: 0.852 | Test Loss: 0.504 | Test AUC 0.889 (6924.21 s) 
Epoch 200, Step 380 | Training Acc: 0.951 | Test Acc: 0.855 | Test Loss: 0.432 | Test AUC 0.900 (6930.97 s) 
Epoch 200, Step 570 | Training Acc: 0.947 | Test Acc: 0.859 | Test Loss: 0.426 | Test AUC 0.892 (6937.72 s) 
Epoch 200, Step 760 | Training Acc: 0.957 | Test Acc: 0.783 | Test Loss: 0.600 | Test AUC 0.858 (6944.46 s) 
(*) Entering Epoch 201 (6951.810 s)
Epoch 201, Step 190 | Training Acc: 0.961 | Test Acc: 0.852 | Test Loss: 0.442 | Test AUC 0.899 (6958.57 s) 
Epoch 201, Step 380 | Training Acc: 0.955 | Test Acc: 0.879 | Test Loss: 0.344 | Test AUC 0.914 (6965.32 s) 
Epoch 201, Step 570 | Training Acc: 0.945 | Test Acc: 0.891 | Test Loss: 0.347 | Test AUC 0.910 (6972.06 s) 
Epoch 201, Step 760 | Training Acc: 0.947 | Test Acc: 0.863 | Test Loss: 0.510 | Test AUC 0.895 (6978.82 s) 
(*) Entering Epoch 202 (6986.136 s)
Epoch 202, Step 190 | Training Acc: 0.957 | Test Acc: 0.893 | Test Loss: 0.301 | Test AUC 0.914 (6992.89 s) 
Epoch 202, Step 380 | Training Acc: 0.926 | Test Acc: 0.895 | Test Loss: 0.339 | Test AUC 0.906 (6999.63 s) 
Epoch 202, Step 570 | Training Acc: 0.951 | Test Acc: 0.861 | Test Loss: 0.433 | Test AUC 0.900 (7006.43 s) 
Epoch 202, Step 760 | Training Acc: 0.936 | Test Acc: 0.850 | Test Loss: 0.399 | Test AUC 0.882 (7013.18 s) 
(*) Entering Epoch 203 (7020.505 s)
Epoch 203, Step 190 | Training Acc: 0.951 | Test Acc: 0.861 | Test Loss: 0.435 | Test AUC 0.899 (7027.26 s) 
Epoch 203, Step 380 | Training Acc: 0.934 | Test Acc: 0.893 | Test Loss: 0.332 | Test AUC 0.909 (7034.05 s) 
Epoch 203, Step 570 | Training Acc: 0.951 | Test Acc: 0.854 | Test Loss: 0.448 | Test AUC 0.898 (7040.80 s) 
Epoch 203, Step 760 | Training Acc: 0.957 | Test Acc: 0.854 | Test Loss: 0.471 | Test AUC 0.893 (7047.56 s) 
(*) Entering Epoch 204 (7054.901 s)
Epoch 204, Step 190 | Training Acc: 0.959 | Test Acc: 0.863 | Test Loss: 0.415 | Test AUC 0.908 (7061.66 s) 
Epoch 204, Step 380 | Training Acc: 0.963 | Test Acc: 0.865 | Test Loss: 0.396 | Test AUC 0.911 (7068.43 s) 
Epoch 204, Step 570 | Training Acc: 0.963 | Test Acc: 0.844 | Test Loss: 0.522 | Test AUC 0.898 (7075.20 s) 
Epoch 204, Step 760 | Training Acc: 0.961 | Test Acc: 0.861 | Test Loss: 0.408 | Test AUC 0.909 (7081.99 s) 
(*) Entering Epoch 205 (7089.305 s)
Epoch 205, Step 190 | Training Acc: 0.957 | Test Acc: 0.887 | Test Loss: 0.377 | Test AUC 0.916 (7096.08 s) 
Epoch 205, Step 380 | Training Acc: 0.975 | Test Acc: 0.889 | Test Loss: 0.305 | Test AUC 0.929 (7102.87 s) 
Epoch 205, Step 570 | Training Acc: 0.965 | Test Acc: 0.830 | Test Loss: 0.456 | Test AUC 0.886 (7109.65 s) 
Epoch 205, Step 760 | Training Acc: 0.939 | Test Acc: 0.838 | Test Loss: 0.493 | Test AUC 0.875 (7116.43 s) 
(*) Entering Epoch 206 (7123.745 s)
Epoch 206, Step 190 | Training Acc: 0.936 | Test Acc: 0.877 | Test Loss: 0.372 | Test AUC 0.901 (7130.50 s) 
Epoch 206, Step 380 | Training Acc: 0.953 | Test Acc: 0.867 | Test Loss: 0.438 | Test AUC 0.903 (7137.30 s) 
Epoch 206, Step 570 | Training Acc: 0.961 | Test Acc: 0.877 | Test Loss: 0.400 | Test AUC 0.914 (7144.07 s) 
Epoch 206, Step 760 | Training Acc: 0.941 | Test Acc: 0.867 | Test Loss: 0.381 | Test AUC 0.896 (7150.84 s) 
(*) Entering Epoch 207 (7158.167 s)
Epoch 207, Step 190 | Training Acc: 0.951 | Test Acc: 0.891 | Test Loss: 0.313 | Test AUC 0.914 (7164.94 s) 
Epoch 207, Step 380 | Training Acc: 0.963 | Test Acc: 0.877 | Test Loss: 0.402 | Test AUC 0.910 (7171.69 s) 
Epoch 207, Step 570 | Training Acc: 0.965 | Test Acc: 0.846 | Test Loss: 0.419 | Test AUC 0.898 (7178.46 s) 
Epoch 207, Step 760 | Training Acc: 0.959 | Test Acc: 0.863 | Test Loss: 0.386 | Test AUC 0.902 (7185.21 s) 
(*) Entering Epoch 208 (7192.535 s)
Epoch 208, Step 190 | Training Acc: 0.949 | Test Acc: 0.877 | Test Loss: 0.405 | Test AUC 0.901 (7199.33 s) 
Epoch 208, Step 380 | Training Acc: 0.971 | Test Acc: 0.869 | Test Loss: 0.404 | Test AUC 0.914 (7206.07 s) 
Epoch 208, Step 570 | Training Acc: 0.977 | Test Acc: 0.873 | Test Loss: 0.413 | Test AUC 0.918 (7212.82 s) 
Epoch 208, Step 760 | Training Acc: 0.943 | Test Acc: 0.781 | Test Loss: 0.660 | Test AUC 0.845 (7219.59 s) 
(*) Entering Epoch 209 (7226.930 s)
Epoch 209, Step 190 | Training Acc: 0.959 | Test Acc: 0.877 | Test Loss: 0.331 | Test AUC 0.910 (7233.67 s) 
Epoch 209, Step 380 | Training Acc: 0.943 | Test Acc: 0.801 | Test Loss: 0.556 | Test AUC 0.858 (7240.44 s) 
Epoch 209, Step 570 | Training Acc: 0.955 | Test Acc: 0.852 | Test Loss: 0.442 | Test AUC 0.895 (7247.21 s) 
Epoch 209, Step 760 | Training Acc: 0.941 | Test Acc: 0.822 | Test Loss: 0.513 | Test AUC 0.866 (7253.95 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch210.ckpt-210
(*) Entering Epoch 210 (7262.291 s)
Epoch 210, Step 190 | Training Acc: 0.951 | Test Acc: 0.850 | Test Loss: 0.448 | Test AUC 0.888 (7269.07 s) 
Epoch 210, Step 380 | Training Acc: 0.971 | Test Acc: 0.875 | Test Loss: 0.361 | Test AUC 0.917 (7275.84 s) 
Epoch 210, Step 570 | Training Acc: 0.961 | Test Acc: 0.840 | Test Loss: 0.549 | Test AUC 0.882 (7282.59 s) 
Epoch 210, Step 760 | Training Acc: 0.961 | Test Acc: 0.852 | Test Loss: 0.479 | Test AUC 0.899 (7289.34 s) 
(*) Entering Epoch 211 (7296.703 s)
Epoch 211, Step 190 | Training Acc: 0.975 | Test Acc: 0.895 | Test Loss: 0.304 | Test AUC 0.929 (7303.45 s) 
Epoch 211, Step 380 | Training Acc: 0.951 | Test Acc: 0.846 | Test Loss: 0.415 | Test AUC 0.893 (7310.20 s) 
Epoch 211, Step 570 | Training Acc: 0.943 | Test Acc: 0.885 | Test Loss: 0.298 | Test AUC 0.906 (7316.99 s) 
Epoch 211, Step 760 | Training Acc: 0.947 | Test Acc: 0.891 | Test Loss: 0.280 | Test AUC 0.911 (7323.72 s) 
(*) Entering Epoch 212 (7331.041 s)
Epoch 212, Step 190 | Training Acc: 0.967 | Test Acc: 0.850 | Test Loss: 0.500 | Test AUC 0.895 (7337.78 s) 
Epoch 212, Step 380 | Training Acc: 0.959 | Test Acc: 0.818 | Test Loss: 0.516 | Test AUC 0.869 (7344.55 s) 
Epoch 212, Step 570 | Training Acc: 0.959 | Test Acc: 0.840 | Test Loss: 0.466 | Test AUC 0.884 (7351.30 s) 
Epoch 212, Step 760 | Training Acc: 0.961 | Test Acc: 0.832 | Test Loss: 0.512 | Test AUC 0.886 (7358.06 s) 
(*) Entering Epoch 213 (7365.373 s)
Epoch 213, Step 190 | Training Acc: 0.945 | Test Acc: 0.891 | Test Loss: 0.373 | Test AUC 0.910 (7372.14 s) 
Epoch 213, Step 380 | Training Acc: 0.959 | Test Acc: 0.883 | Test Loss: 0.393 | Test AUC 0.918 (7378.92 s) 
Epoch 213, Step 570 | Training Acc: 0.963 | Test Acc: 0.859 | Test Loss: 0.397 | Test AUC 0.907 (7385.69 s) 
Epoch 213, Step 760 | Training Acc: 0.959 | Test Acc: 0.896 | Test Loss: 0.333 | Test AUC 0.924 (7392.44 s) 
(*) Entering Epoch 214 (7399.799 s)
Epoch 214, Step 190 | Training Acc: 0.938 | Test Acc: 0.871 | Test Loss: 0.406 | Test AUC 0.891 (7406.57 s) 
Epoch 214, Step 380 | Training Acc: 0.945 | Test Acc: 0.865 | Test Loss: 0.371 | Test AUC 0.896 (7413.35 s) 
Epoch 214, Step 570 | Training Acc: 0.953 | Test Acc: 0.879 | Test Loss: 0.372 | Test AUC 0.910 (7420.13 s) 
Epoch 214, Step 760 | Training Acc: 0.957 | Test Acc: 0.855 | Test Loss: 0.409 | Test AUC 0.899 (7426.89 s) 
(*) Entering Epoch 215 (7434.232 s)
Epoch 215, Step 190 | Training Acc: 0.959 | Test Acc: 0.875 | Test Loss: 0.410 | Test AUC 0.914 (7441.02 s) 
Epoch 215, Step 380 | Training Acc: 0.953 | Test Acc: 0.865 | Test Loss: 0.416 | Test AUC 0.900 (7447.77 s) 
Epoch 215, Step 570 | Training Acc: 0.971 | Test Acc: 0.881 | Test Loss: 0.417 | Test AUC 0.921 (7454.54 s) 
Epoch 215, Step 760 | Training Acc: 0.963 | Test Acc: 0.865 | Test Loss: 0.426 | Test AUC 0.910 (7461.30 s) 
(*) Entering Epoch 216 (7468.671 s)
Epoch 216, Step 190 | Training Acc: 0.953 | Test Acc: 0.826 | Test Loss: 0.524 | Test AUC 0.877 (7475.44 s) 
Epoch 216, Step 380 | Training Acc: 0.967 | Test Acc: 0.854 | Test Loss: 0.461 | Test AUC 0.901 (7482.20 s) 
Epoch 216, Step 570 | Training Acc: 0.969 | Test Acc: 0.896 | Test Loss: 0.337 | Test AUC 0.929 (7488.94 s) 
Epoch 216, Step 760 | Training Acc: 0.949 | Test Acc: 0.861 | Test Loss: 0.354 | Test AUC 0.898 (7495.70 s) 
(*) Entering Epoch 217 (7503.030 s)
Epoch 217, Step 190 | Training Acc: 0.963 | Test Acc: 0.877 | Test Loss: 0.332 | Test AUC 0.912 (7509.82 s) 
Epoch 217, Step 380 | Training Acc: 0.951 | Test Acc: 0.879 | Test Loss: 0.390 | Test AUC 0.912 (7516.58 s) 
Epoch 217, Step 570 | Training Acc: 0.963 | Test Acc: 0.811 | Test Loss: 0.582 | Test AUC 0.862 (7523.36 s) 
Epoch 217, Step 760 | Training Acc: 0.928 | Test Acc: 0.809 | Test Loss: 0.545 | Test AUC 0.842 (7530.14 s) 
(*) Entering Epoch 218 (7537.485 s)
Epoch 218, Step 190 | Training Acc: 0.957 | Test Acc: 0.834 | Test Loss: 0.581 | Test AUC 0.880 (7544.24 s) 
Epoch 218, Step 380 | Training Acc: 0.953 | Test Acc: 0.848 | Test Loss: 0.484 | Test AUC 0.888 (7551.04 s) 
Epoch 218, Step 570 | Training Acc: 0.955 | Test Acc: 0.846 | Test Loss: 0.466 | Test AUC 0.890 (7557.84 s) 
Epoch 218, Step 760 | Training Acc: 0.973 | Test Acc: 0.873 | Test Loss: 0.360 | Test AUC 0.917 (7564.60 s) 
(*) Entering Epoch 219 (7571.929 s)
Epoch 219, Step 190 | Training Acc: 0.939 | Test Acc: 0.852 | Test Loss: 0.462 | Test AUC 0.878 (7578.69 s) 
Epoch 219, Step 380 | Training Acc: 0.961 | Test Acc: 0.834 | Test Loss: 0.548 | Test AUC 0.886 (7585.46 s) 
Epoch 219, Step 570 | Training Acc: 0.949 | Test Acc: 0.803 | Test Loss: 0.620 | Test AUC 0.857 (7592.23 s) 
Epoch 219, Step 760 | Training Acc: 0.951 | Test Acc: 0.842 | Test Loss: 0.464 | Test AUC 0.882 (7599.00 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch220.ckpt-220
(*) Entering Epoch 220 (7607.317 s)
Epoch 220, Step 190 | Training Acc: 0.949 | Test Acc: 0.869 | Test Loss: 0.441 | Test AUC 0.903 (7614.09 s) 
Epoch 220, Step 380 | Training Acc: 0.947 | Test Acc: 0.873 | Test Loss: 0.368 | Test AUC 0.907 (7620.92 s) 
Epoch 220, Step 570 | Training Acc: 0.967 | Test Acc: 0.811 | Test Loss: 0.500 | Test AUC 0.876 (7627.67 s) 
Epoch 220, Step 760 | Training Acc: 0.949 | Test Acc: 0.850 | Test Loss: 0.415 | Test AUC 0.891 (7634.42 s) 
(*) Entering Epoch 221 (7641.760 s)
Epoch 221, Step 190 | Training Acc: 0.957 | Test Acc: 0.875 | Test Loss: 0.411 | Test AUC 0.910 (7648.53 s) 
Epoch 221, Step 380 | Training Acc: 0.959 | Test Acc: 0.861 | Test Loss: 0.394 | Test AUC 0.903 (7655.29 s) 
Epoch 221, Step 570 | Training Acc: 0.955 | Test Acc: 0.859 | Test Loss: 0.395 | Test AUC 0.897 (7662.04 s) 
Epoch 221, Step 760 | Training Acc: 0.949 | Test Acc: 0.844 | Test Loss: 0.429 | Test AUC 0.891 (7668.80 s) 
(*) Entering Epoch 222 (7676.117 s)
Epoch 222, Step 190 | Training Acc: 0.957 | Test Acc: 0.844 | Test Loss: 0.508 | Test AUC 0.889 (7682.88 s) 
Epoch 222, Step 380 | Training Acc: 0.967 | Test Acc: 0.855 | Test Loss: 0.424 | Test AUC 0.902 (7689.63 s) 
Epoch 222, Step 570 | Training Acc: 0.943 | Test Acc: 0.871 | Test Loss: 0.382 | Test AUC 0.901 (7696.39 s) 
Epoch 222, Step 760 | Training Acc: 0.957 | Test Acc: 0.869 | Test Loss: 0.399 | Test AUC 0.907 (7703.12 s) 
(*) Entering Epoch 223 (7710.449 s)
Epoch 223, Step 190 | Training Acc: 0.969 | Test Acc: 0.812 | Test Loss: 0.536 | Test AUC 0.875 (7717.20 s) 
Epoch 223, Step 380 | Training Acc: 0.963 | Test Acc: 0.848 | Test Loss: 0.458 | Test AUC 0.888 (7723.96 s) 
Epoch 223, Step 570 | Training Acc: 0.951 | Test Acc: 0.838 | Test Loss: 0.522 | Test AUC 0.885 (7730.71 s) 
Epoch 223, Step 760 | Training Acc: 0.967 | Test Acc: 0.859 | Test Loss: 0.451 | Test AUC 0.903 (7737.49 s) 
(*) Entering Epoch 224 (7744.821 s)
Epoch 224, Step 190 | Training Acc: 0.957 | Test Acc: 0.867 | Test Loss: 0.375 | Test AUC 0.908 (7751.57 s) 
Epoch 224, Step 380 | Training Acc: 0.949 | Test Acc: 0.875 | Test Loss: 0.366 | Test AUC 0.907 (7758.32 s) 
Epoch 224, Step 570 | Training Acc: 0.965 | Test Acc: 0.881 | Test Loss: 0.320 | Test AUC 0.918 (7765.06 s) 
Epoch 224, Step 760 | Training Acc: 0.961 | Test Acc: 0.816 | Test Loss: 0.555 | Test AUC 0.875 (7771.83 s) 
(*) Entering Epoch 225 (7779.157 s)
Epoch 225, Step 190 | Training Acc: 0.945 | Test Acc: 0.838 | Test Loss: 0.418 | Test AUC 0.878 (7785.94 s) 
Epoch 225, Step 380 | Training Acc: 0.965 | Test Acc: 0.865 | Test Loss: 0.411 | Test AUC 0.906 (7792.68 s) 
Epoch 225, Step 570 | Training Acc: 0.969 | Test Acc: 0.846 | Test Loss: 0.486 | Test AUC 0.899 (7799.44 s) 
Epoch 225, Step 760 | Training Acc: 0.941 | Test Acc: 0.834 | Test Loss: 0.625 | Test AUC 0.877 (7806.19 s) 
(*) Entering Epoch 226 (7813.496 s)
Epoch 226, Step 190 | Training Acc: 0.969 | Test Acc: 0.848 | Test Loss: 0.467 | Test AUC 0.898 (7820.27 s) 
Epoch 226, Step 380 | Training Acc: 0.951 | Test Acc: 0.855 | Test Loss: 0.458 | Test AUC 0.889 (7827.03 s) 
Epoch 226, Step 570 | Training Acc: 0.953 | Test Acc: 0.885 | Test Loss: 0.365 | Test AUC 0.912 (7833.79 s) 
Epoch 226, Step 760 | Training Acc: 0.951 | Test Acc: 0.850 | Test Loss: 0.501 | Test AUC 0.896 (7840.58 s) 
(*) Entering Epoch 227 (7847.904 s)
Epoch 227, Step 190 | Training Acc: 0.963 | Test Acc: 0.895 | Test Loss: 0.388 | Test AUC 0.922 (7854.66 s) 
Epoch 227, Step 380 | Training Acc: 0.979 | Test Acc: 0.861 | Test Loss: 0.449 | Test AUC 0.911 (7861.43 s) 
Epoch 227, Step 570 | Training Acc: 0.957 | Test Acc: 0.895 | Test Loss: 0.383 | Test AUC 0.919 (7868.20 s) 
Epoch 227, Step 760 | Training Acc: 0.951 | Test Acc: 0.893 | Test Loss: 0.359 | Test AUC 0.917 (7874.95 s) 
(*) Entering Epoch 228 (7882.273 s)
Epoch 228, Step 190 | Training Acc: 0.957 | Test Acc: 0.881 | Test Loss: 0.466 | Test AUC 0.911 (7889.05 s) 
Epoch 228, Step 380 | Training Acc: 0.953 | Test Acc: 0.877 | Test Loss: 0.490 | Test AUC 0.909 (7895.81 s) 
Epoch 228, Step 570 | Training Acc: 0.936 | Test Acc: 0.857 | Test Loss: 0.409 | Test AUC 0.886 (7902.57 s) 
Epoch 228, Step 760 | Training Acc: 0.961 | Test Acc: 0.861 | Test Loss: 0.423 | Test AUC 0.897 (7909.32 s) 
(*) Entering Epoch 229 (7916.615 s)
Epoch 229, Step 190 | Training Acc: 0.975 | Test Acc: 0.863 | Test Loss: 0.420 | Test AUC 0.910 (7923.36 s) 
Epoch 229, Step 380 | Training Acc: 0.951 | Test Acc: 0.871 | Test Loss: 0.396 | Test AUC 0.905 (7930.10 s) 
Epoch 229, Step 570 | Training Acc: 0.957 | Test Acc: 0.865 | Test Loss: 0.420 | Test AUC 0.905 (7936.87 s) 
Epoch 229, Step 760 | Training Acc: 0.957 | Test Acc: 0.893 | Test Loss: 0.342 | Test AUC 0.920 (7943.62 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch230.ckpt-230
(*) Entering Epoch 230 (7951.933 s)
Epoch 230, Step 190 | Training Acc: 0.947 | Test Acc: 0.898 | Test Loss: 0.280 | Test AUC 0.918 (7958.70 s) 
Epoch 230, Step 380 | Training Acc: 0.945 | Test Acc: 0.854 | Test Loss: 0.396 | Test AUC 0.890 (7965.46 s) 
Epoch 230, Step 570 | Training Acc: 0.945 | Test Acc: 0.863 | Test Loss: 0.386 | Test AUC 0.898 (7972.22 s) 
Epoch 230, Step 760 | Training Acc: 0.959 | Test Acc: 0.887 | Test Loss: 0.343 | Test AUC 0.919 (7979.01 s) 
(*) Entering Epoch 231 (7986.374 s)
Epoch 231, Step 190 | Training Acc: 0.955 | Test Acc: 0.852 | Test Loss: 0.519 | Test AUC 0.897 (7993.13 s) 
Epoch 231, Step 380 | Training Acc: 0.961 | Test Acc: 0.828 | Test Loss: 0.464 | Test AUC 0.877 (7999.90 s) 
Epoch 231, Step 570 | Training Acc: 0.955 | Test Acc: 0.848 | Test Loss: 0.477 | Test AUC 0.890 (8006.66 s) 
Epoch 231, Step 760 | Training Acc: 0.963 | Test Acc: 0.816 | Test Loss: 0.576 | Test AUC 0.874 (8013.45 s) 
(*) Entering Epoch 232 (8020.809 s)
Epoch 232, Step 190 | Training Acc: 0.957 | Test Acc: 0.844 | Test Loss: 0.453 | Test AUC 0.890 (8027.56 s) 
Epoch 232, Step 380 | Training Acc: 0.951 | Test Acc: 0.838 | Test Loss: 0.526 | Test AUC 0.882 (8034.32 s) 
Epoch 232, Step 570 | Training Acc: 0.951 | Test Acc: 0.832 | Test Loss: 0.544 | Test AUC 0.882 (8041.16 s) 
Epoch 232, Step 760 | Training Acc: 0.951 | Test Acc: 0.863 | Test Loss: 0.436 | Test AUC 0.897 (8047.93 s) 
(*) Entering Epoch 233 (8055.246 s)
Epoch 233, Step 190 | Training Acc: 0.938 | Test Acc: 0.881 | Test Loss: 0.353 | Test AUC 0.906 (8062.00 s) 
Epoch 233, Step 380 | Training Acc: 0.947 | Test Acc: 0.871 | Test Loss: 0.444 | Test AUC 0.905 (8068.76 s) 
Epoch 233, Step 570 | Training Acc: 0.961 | Test Acc: 0.848 | Test Loss: 0.501 | Test AUC 0.899 (8075.52 s) 
Epoch 233, Step 760 | Training Acc: 0.961 | Test Acc: 0.848 | Test Loss: 0.505 | Test AUC 0.897 (8082.28 s) 
(*) Entering Epoch 234 (8089.648 s)
Epoch 234, Step 190 | Training Acc: 0.965 | Test Acc: 0.854 | Test Loss: 0.439 | Test AUC 0.903 (8096.41 s) 
Epoch 234, Step 380 | Training Acc: 0.965 | Test Acc: 0.887 | Test Loss: 0.380 | Test AUC 0.922 (8103.17 s) 
Epoch 234, Step 570 | Training Acc: 0.961 | Test Acc: 0.879 | Test Loss: 0.429 | Test AUC 0.915 (8109.92 s) 
Epoch 234, Step 760 | Training Acc: 0.961 | Test Acc: 0.844 | Test Loss: 0.410 | Test AUC 0.887 (8116.67 s) 
(*) Entering Epoch 235 (8123.989 s)
Epoch 235, Step 190 | Training Acc: 0.955 | Test Acc: 0.883 | Test Loss: 0.334 | Test AUC 0.913 (8130.77 s) 
Epoch 235, Step 380 | Training Acc: 0.961 | Test Acc: 0.898 | Test Loss: 0.339 | Test AUC 0.921 (8137.51 s) 
Epoch 235, Step 570 | Training Acc: 0.938 | Test Acc: 0.871 | Test Loss: 0.405 | Test AUC 0.895 (8144.26 s) 
Epoch 235, Step 760 | Training Acc: 0.941 | Test Acc: 0.883 | Test Loss: 0.436 | Test AUC 0.903 (8151.02 s) 
(*) Entering Epoch 236 (8158.361 s)
Epoch 236, Step 190 | Training Acc: 0.953 | Test Acc: 0.885 | Test Loss: 0.370 | Test AUC 0.913 (8165.11 s) 
Epoch 236, Step 380 | Training Acc: 0.953 | Test Acc: 0.873 | Test Loss: 0.370 | Test AUC 0.910 (8171.89 s) 
Epoch 236, Step 570 | Training Acc: 0.961 | Test Acc: 0.863 | Test Loss: 0.379 | Test AUC 0.909 (8178.66 s) 
Epoch 236, Step 760 | Training Acc: 0.924 | Test Acc: 0.848 | Test Loss: 0.482 | Test AUC 0.870 (8185.44 s) 
(*) Entering Epoch 237 (8192.777 s)
Epoch 237, Step 190 | Training Acc: 0.967 | Test Acc: 0.789 | Test Loss: 0.621 | Test AUC 0.858 (8199.55 s) 
Epoch 237, Step 380 | Training Acc: 0.959 | Test Acc: 0.846 | Test Loss: 0.465 | Test AUC 0.889 (8206.31 s) 
Epoch 237, Step 570 | Training Acc: 0.938 | Test Acc: 0.887 | Test Loss: 0.380 | Test AUC 0.900 (8213.06 s) 
Epoch 237, Step 760 | Training Acc: 0.957 | Test Acc: 0.859 | Test Loss: 0.431 | Test AUC 0.901 (8219.82 s) 
(*) Entering Epoch 238 (8227.182 s)
Epoch 238, Step 190 | Training Acc: 0.955 | Test Acc: 0.871 | Test Loss: 0.358 | Test AUC 0.908 (8233.95 s) 
Epoch 238, Step 380 | Training Acc: 0.971 | Test Acc: 0.863 | Test Loss: 0.415 | Test AUC 0.913 (8240.72 s) 
Epoch 238, Step 570 | Training Acc: 0.947 | Test Acc: 0.893 | Test Loss: 0.264 | Test AUC 0.913 (8247.48 s) 
Epoch 238, Step 760 | Training Acc: 0.965 | Test Acc: 0.811 | Test Loss: 0.535 | Test AUC 0.870 (8254.23 s) 
(*) Entering Epoch 239 (8261.563 s)
Epoch 239, Step 190 | Training Acc: 0.949 | Test Acc: 0.807 | Test Loss: 0.673 | Test AUC 0.859 (8268.33 s) 
Epoch 239, Step 380 | Training Acc: 0.939 | Test Acc: 0.852 | Test Loss: 0.403 | Test AUC 0.885 (8275.14 s) 
Epoch 239, Step 570 | Training Acc: 0.955 | Test Acc: 0.879 | Test Loss: 0.446 | Test AUC 0.910 (8281.90 s) 
Epoch 239, Step 760 | Training Acc: 0.959 | Test Acc: 0.871 | Test Loss: 0.382 | Test AUC 0.911 (8288.69 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch240.ckpt-240
(*) Entering Epoch 240 (8297.068 s)
Epoch 240, Step 190 | Training Acc: 0.959 | Test Acc: 0.871 | Test Loss: 0.407 | Test AUC 0.907 (8303.85 s) 
Epoch 240, Step 380 | Training Acc: 0.967 | Test Acc: 0.877 | Test Loss: 0.332 | Test AUC 0.918 (8310.63 s) 
Epoch 240, Step 570 | Training Acc: 0.934 | Test Acc: 0.869 | Test Loss: 0.414 | Test AUC 0.895 (8317.40 s) 
Epoch 240, Step 760 | Training Acc: 0.949 | Test Acc: 0.840 | Test Loss: 0.508 | Test AUC 0.878 (8324.15 s) 
(*) Entering Epoch 241 (8331.478 s)
Epoch 241, Step 190 | Training Acc: 0.951 | Test Acc: 0.840 | Test Loss: 0.500 | Test AUC 0.881 (8338.25 s) 
Epoch 241, Step 380 | Training Acc: 0.959 | Test Acc: 0.783 | Test Loss: 0.724 | Test AUC 0.854 (8345.03 s) 
Epoch 241, Step 570 | Training Acc: 0.963 | Test Acc: 0.832 | Test Loss: 0.524 | Test AUC 0.887 (8351.80 s) 
Epoch 241, Step 760 | Training Acc: 0.943 | Test Acc: 0.830 | Test Loss: 0.455 | Test AUC 0.873 (8358.58 s) 
(*) Entering Epoch 242 (8365.904 s)
Epoch 242, Step 190 | Training Acc: 0.959 | Test Acc: 0.838 | Test Loss: 0.484 | Test AUC 0.888 (8372.66 s) 
Epoch 242, Step 380 | Training Acc: 0.961 | Test Acc: 0.832 | Test Loss: 0.508 | Test AUC 0.883 (8379.44 s) 
Epoch 242, Step 570 | Training Acc: 0.953 | Test Acc: 0.863 | Test Loss: 0.471 | Test AUC 0.899 (8386.20 s) 
Epoch 242, Step 760 | Training Acc: 0.965 | Test Acc: 0.881 | Test Loss: 0.347 | Test AUC 0.914 (8393.02 s) 
(*) Entering Epoch 243 (8400.371 s)
Epoch 243, Step 190 | Training Acc: 0.967 | Test Acc: 0.895 | Test Loss: 0.355 | Test AUC 0.925 (8407.14 s) 
Epoch 243, Step 380 | Training Acc: 0.957 | Test Acc: 0.852 | Test Loss: 0.422 | Test AUC 0.887 (8413.89 s) 
Epoch 243, Step 570 | Training Acc: 0.953 | Test Acc: 0.830 | Test Loss: 0.485 | Test AUC 0.881 (8420.63 s) 
Epoch 243, Step 760 | Training Acc: 0.951 | Test Acc: 0.869 | Test Loss: 0.390 | Test AUC 0.903 (8427.39 s) 
(*) Entering Epoch 244 (8434.707 s)
Epoch 244, Step 190 | Training Acc: 0.951 | Test Acc: 0.865 | Test Loss: 0.393 | Test AUC 0.901 (8441.45 s) 
Epoch 244, Step 380 | Training Acc: 0.930 | Test Acc: 0.871 | Test Loss: 0.368 | Test AUC 0.896 (8448.24 s) 
Epoch 244, Step 570 | Training Acc: 0.971 | Test Acc: 0.789 | Test Loss: 0.628 | Test AUC 0.864 (8455.00 s) 
Epoch 244, Step 760 | Training Acc: 0.953 | Test Acc: 0.848 | Test Loss: 0.382 | Test AUC 0.890 (8461.85 s) 
(*) Entering Epoch 245 (8469.170 s)
Epoch 245, Step 190 | Training Acc: 0.947 | Test Acc: 0.900 | Test Loss: 0.314 | Test AUC 0.920 (8475.95 s) 
Epoch 245, Step 380 | Training Acc: 0.967 | Test Acc: 0.906 | Test Loss: 0.267 | Test AUC 0.930 (8482.75 s) 
Epoch 245, Step 570 | Training Acc: 0.957 | Test Acc: 0.902 | Test Loss: 0.268 | Test AUC 0.923 (8489.53 s) 
Epoch 245, Step 760 | Training Acc: 0.947 | Test Acc: 0.861 | Test Loss: 0.436 | Test AUC 0.899 (8496.34 s) 
(*) Entering Epoch 246 (8503.663 s)
Epoch 246, Step 190 | Training Acc: 0.951 | Test Acc: 0.838 | Test Loss: 0.493 | Test AUC 0.883 (8510.44 s) 
Epoch 246, Step 380 | Training Acc: 0.973 | Test Acc: 0.857 | Test Loss: 0.470 | Test AUC 0.908 (8517.24 s) 
Epoch 246, Step 570 | Training Acc: 0.963 | Test Acc: 0.848 | Test Loss: 0.439 | Test AUC 0.901 (8523.98 s) 
Epoch 246, Step 760 | Training Acc: 0.955 | Test Acc: 0.787 | Test Loss: 0.601 | Test AUC 0.850 (8530.75 s) 
(*) Entering Epoch 247 (8538.061 s)
Epoch 247, Step 190 | Training Acc: 0.955 | Test Acc: 0.873 | Test Loss: 0.405 | Test AUC 0.903 (8544.88 s) 
Epoch 247, Step 380 | Training Acc: 0.949 | Test Acc: 0.857 | Test Loss: 0.494 | Test AUC 0.892 (8551.70 s) 
Epoch 247, Step 570 | Training Acc: 0.953 | Test Acc: 0.871 | Test Loss: 0.365 | Test AUC 0.906 (8558.44 s) 
Epoch 247, Step 760 | Training Acc: 0.959 | Test Acc: 0.857 | Test Loss: 0.424 | Test AUC 0.899 (8565.22 s) 
(*) Entering Epoch 248 (8572.534 s)
Epoch 248, Step 190 | Training Acc: 0.969 | Test Acc: 0.863 | Test Loss: 0.405 | Test AUC 0.906 (8579.31 s) 
Epoch 248, Step 380 | Training Acc: 0.941 | Test Acc: 0.869 | Test Loss: 0.394 | Test AUC 0.900 (8586.06 s) 
Epoch 248, Step 570 | Training Acc: 0.967 | Test Acc: 0.854 | Test Loss: 0.520 | Test AUC 0.905 (8592.83 s) 
Epoch 248, Step 760 | Training Acc: 0.965 | Test Acc: 0.812 | Test Loss: 0.648 | Test AUC 0.872 (8599.57 s) 
(*) Entering Epoch 249 (8606.908 s)
Epoch 249, Step 190 | Training Acc: 0.969 | Test Acc: 0.854 | Test Loss: 0.512 | Test AUC 0.903 (8613.68 s) 
Epoch 249, Step 380 | Training Acc: 0.959 | Test Acc: 0.873 | Test Loss: 0.409 | Test AUC 0.910 (8620.44 s) 
Epoch 249, Step 570 | Training Acc: 0.959 | Test Acc: 0.844 | Test Loss: 0.468 | Test AUC 0.896 (8627.23 s) 
Epoch 249, Step 760 | Training Acc: 0.949 | Test Acc: 0.885 | Test Loss: 0.353 | Test AUC 0.912 (8634.04 s) 
Graph saved to file: checkpoints/vDNN_kst_rho0_epoch250.ckpt-250
(*) Entering Epoch 250 (8642.400 s)
Epoch 250, Step 190 | Training Acc: 0.955 | Test Acc: 0.898 | Test Loss: 0.319 | Test AUC 0.922 (8649.16 s) 
Epoch 250, Step 380 | Training Acc: 0.965 | Test Acc: 0.832 | Test Loss: 0.539 | Test AUC 0.883 (8655.94 s) 
Epoch 250, Step 570 | Training Acc: 0.969 | Test Acc: 0.826 | Test Loss: 0.556 | Test AUC 0.885 (8662.73 s) 
Epoch 250, Step 760 | Training Acc: 0.955 | Test Acc: 0.838 | Test Loss: 0.485 | Test AUC 0.888 (8669.75 s) 
(*) Entering Epoch 251 (8677.076 s)
Epoch 251, Step 190 | Training Acc: 0.963 | Test Acc: 0.816 | Test Loss: 0.587 | Test AUC 0.868 (8683.80 s) 
Epoch 251, Step 380 | Training Acc: 0.967 | Test Acc: 0.850 | Test Loss: 0.365 | Test AUC 0.894 (8690.58 s) 
Epoch 251, Step 570 | Training Acc: 0.953 | Test Acc: 0.877 | Test Loss: 0.417 | Test AUC 0.904 (8697.36 s) 
Epoch 251, Step 760 | Training Acc: 0.973 | Test Acc: 0.889 | Test Loss: 0.406 | Test AUC 0.925 (8704.15 s) 
(*) Entering Epoch 252 (8711.508 s)
Epoch 252, Step 190 | Training Acc: 0.949 | Test Acc: 0.852 | Test Loss: 0.516 | Test AUC 0.892 (8718.28 s) 
Epoch 252, Step 380 | Training Acc: 0.971 | Test Acc: 0.844 | Test Loss: 0.545 | Test AUC 0.900 (8725.09 s) 
Epoch 252, Step 570 | Training Acc: 0.957 | Test Acc: 0.826 | Test Loss: 0.539 | Test AUC 0.875 (8731.88 s) 
Epoch 252, Step 760 | Training Acc: 0.957 | Test Acc: 0.854 | Test Loss: 0.468 | Test AUC 0.899 (8738.65 s) 
(*) Entering Epoch 253 (8745.964 s)
Epoch 253, Step 190 | Training Acc: 0.959 | Test Acc: 0.836 | Test Loss: 0.584 | Test AUC 0.886 (8752.73 s) 
Epoch 253, Step 380 | Training Acc: 0.963 | Test Acc: 0.867 | Test Loss: 0.398 | Test AUC 0.908 (8759.52 s) 
Epoch 253, Step 570 | Training Acc: 0.941 | Test Acc: 0.861 | Test Loss: 0.445 | Test AUC 0.897 (8766.36 s) 
Epoch 253, Step 760 | Training Acc: 0.953 | Test Acc: 0.877 | Test Loss: 0.325 | Test AUC 0.908 (8773.12 s) 
(*) Entering Epoch 254 (8780.488 s)
Epoch 254, Step 190 | Training Acc: 0.943 | Test Acc: 0.848 | Test Loss: 0.442 | Test AUC 0.884 (8787.26 s) 
Epoch 254, Step 380 | Training Acc: 0.945 | Test Acc: 0.875 | Test Loss: 0.376 | Test AUC 0.902 (8794.05 s) 
Epoch 254, Step 570 | Training Acc: 0.959 | Test Acc: 0.893 | Test Loss: 0.307 | Test AUC 0.922 (8800.87 s) 
Epoch 254, Step 760 | Training Acc: 0.959 | Test Acc: 0.848 | Test Loss: 0.462 | Test AUC 0.898 (8807.66 s) 
(*) Entering Epoch 255 (8814.996 s)
Epoch 255, Step 190 | Training Acc: 0.941 | Test Acc: 0.885 | Test Loss: 0.362 | Test AUC 0.912 (8821.84 s) 
Epoch 255, Step 380 | Training Acc: 0.955 | Test Acc: 0.867 | Test Loss: 0.377 | Test AUC 0.905 (8828.63 s) 
Epoch 255, Step 570 | Training Acc: 0.959 | Test Acc: 0.887 | Test Loss: 0.355 | Test AUC 0.916 (8835.39 s) 
Epoch 255, Step 760 | Training Acc: 0.963 | Test Acc: 0.855 | Test Loss: 0.379 | Test AUC 0.900 (8842.20 s) 
Model saved to file: checkpoints/vDNN_kst_rho0_end.ckpt-255
Training Complete. Time elapsed: 8850.642 s
Architecture: rho0 - kst | Layers: 12 | Dropout: 0.75 | Base LR: 0.0008 | Epochs: 256

Making Predictions

Classification on a new instance is given by the softmax of the output of the final readout layer.





In [7]:



    


ckpt = tf.train.get_checkpoint_state(directories.checkpoints)
vDNN = vanillaDNN(config, training = False)
network_output = vDNN.predict(ckpt)

np.save(os.path.join(directories.checkpoints, '{}_{}_y_pred.npy'.format(config.channel, config.mode)), network_output)
np.save(os.path.join(directories.checkpoints, '{}_{}_y_test.npy'.format(config.channel, config.mode)), df_y_test.values)



    


















    






Building SELU architecture
Building SELU architecture
Building SELU architecture
Building SELU architecture
INFO:tensorflow:Restoring parameters from checkpoints/vDNN_kst_rho0_end.ckpt-255
checkpoints/vDNN_kst_rho0_end.ckpt-255 restored.
Validation accuracy: 0.830248
Validation AUC: 0.815689










    






/home/jtan/anaconda3/lib/python3.6/site-packages/matplotlib/font_manager.py:1297: UserWarning: findfont: Font family ['sans-serif'] not found. Falling back to DejaVu Sans
  (prop.get_family(), self.defaultFamily[fontext]))










    






AUC: 0.887453










    

















    






Inference complete. Duration: 4.58042 s

















In [ ]:

Content source: Justin-Tan/hep-analysis

Similar notebooks:

notebook.community | gallery | about