notebook.community

Edit and run 





In [1]:



    


import tensorflow as tf
from tensorflow import data
import numpy as np
import shutil
import math
from datetime import datetime
from tensorflow.python.feature_column import feature_column

from tensorflow.contrib.learn import learn_runner
from tensorflow.contrib.learn import make_export_strategy

print(tf.__version__)



    


















    






/Users/khalidsalama/anaconda/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
  return f(*args, **kwds)










    






1.4.0

Steps to use the TF Experiment APIs

  1. Define dataset metadata
  2. Define data input function to read the data from .tfrecord files + feature processing
  3. Create TF feature columns based on metadata + extended feature columns
  4. Define an a model function with the required feature columns, EstimatorSpecs, & parameters
  5. Run an Experiment with learn_runner to train, evaluate, and export the model
  6. Evaluate the model using test data
  7. Perform predictions & serving the exported model (using CSV/JSON input)




In [2]:



    


MODEL_NAME = 'class-model-02'

TRAIN_DATA_FILES_PATTERN = 'data/train-*.csv'
VALID_DATA_FILES_PATTERN = 'data/valid-*.csv'
TEST_DATA_FILES_PATTERN = 'data/test-*.csv'

RESUME_TRAINING = False
PROCESS_FEATURES = True
EXTEND_FEATURE_COLUMNS = True
MULTI_THREADING = True

1. Define Dataset Metadata

  • tf.example feature names and defaults
  • Numeric and categorical feature names
  • Target feature name
  • Target feature labels
  • Unused features




In [3]:



    


HEADER = ['key','x','y','alpha','beta','target']
HEADER_DEFAULTS = [[0], [0.0], [0.0], ['NA'], ['NA'], ['NA']]

NUMERIC_FEATURE_NAMES = ['x', 'y']  

CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY = {'alpha':['ax01', 'ax02'], 'beta':['bx01', 'bx02']}
CATEGORICAL_FEATURE_NAMES = list(CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.keys())

FEATURE_NAMES = NUMERIC_FEATURE_NAMES + CATEGORICAL_FEATURE_NAMES

TARGET_NAME = 'target'

TARGET_LABELS = ['positive', 'negative']

UNUSED_FEATURE_NAMES = list(set(HEADER) - set(FEATURE_NAMES) - {TARGET_NAME})

print("Header: {}".format(HEADER))
print("Numeric Features: {}".format(NUMERIC_FEATURE_NAMES))
print("Categorical Features: {}".format(CATEGORICAL_FEATURE_NAMES))
print("Target: {} - labels: {}".format(TARGET_NAME, TARGET_LABELS))
print("Unused Features: {}".format(UNUSED_FEATURE_NAMES))



    


















    






Header: ['key', 'x', 'y', 'alpha', 'beta', 'target']
Numeric Features: ['x', 'y']
Categorical Features: ['alpha', 'beta']
Target: target - labels: ['positive', 'negative']
Unused Features: ['key']

2. Define Data Input Function

  • Input csv files name pattern
  • Use TF Dataset APIs to read and process the data
  • Parse CSV lines to feature tensors
  • Apply feature processing
  • Return (features, target) tensors

a. Parsing and preprocessing logic





In [4]:



    


def parse_csv_row(csv_row):
    
    columns = tf.decode_csv(csv_row, record_defaults=HEADER_DEFAULTS)
    features = dict(zip(HEADER, columns))
    
    for column in UNUSED_FEATURE_NAMES:
        features.pop(column)
    
    target = features.pop(TARGET_NAME)

    return features, target

def process_features(features):

    features["x_2"] = tf.square(features['x'])
    features["y_2"] = tf.square(features['y'])
    features["xy"] = tf.multiply(features['x'], features['y']) # features['x'] * features['y']
    features['dist_xy'] =  tf.sqrt(tf.squared_difference(features['x'],features['y']))
    
    return features

b. Data pipeline input function





In [5]:



    


def parse_label_column(label_string_tensor):
    table = tf.contrib.lookup.index_table_from_tensor(tf.constant(TARGET_LABELS))
    return table.lookup(label_string_tensor)

def csv_input_fn(files_name_pattern, mode=tf.estimator.ModeKeys.EVAL, 
                 skip_header_lines=0, 
                 num_epochs=None, 
                 batch_size=200):
    
    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False
    
    print("")
    print("* data input_fn:")
    print("================")
    print("Input file(s): {}".format(files_name_pattern))
    print("Batch size: {}".format(batch_size))
    print("Epoch Count: {}".format(num_epochs))
    print("Mode: {}".format(mode))
    print("Shuffle: {}".format(shuffle))
    print("================")
    print("")

    file_names = tf.matching_files(files_name_pattern)
    dataset = data.TextLineDataset(filenames=file_names)
    
    dataset = dataset.skip(skip_header_lines)
    
    if shuffle:
        dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)
    
    dataset = dataset.batch(batch_size)
    dataset = dataset.map(lambda csv_row: parse_csv_row(csv_row))
    
    if PROCESS_FEATURES:
        dataset = dataset.map(lambda features, target: (process_features(features), target))
        
    dataset = dataset.repeat(num_epochs)
    iterator = dataset.make_one_shot_iterator()
    
    features, target = iterator.get_next()
    return features, parse_label_column(target)



    











In [6]:



    


features, target = csv_input_fn(files_name_pattern="")
print("Feature read from CSV: {}".format(list(features.keys())))
print("Target read from CSV: {}".format(target))



    


















    






* data input_fn:
================
Input file(s): 
Batch size: 200
Epoch Count: None
Mode: eval
Shuffle: False
================

Feature read from CSV: ['x', 'y', 'alpha', 'beta', 'x_2', 'y_2', 'xy', 'dist_xy']
Target read from CSV: Tensor("hash_table_Lookup:0", shape=(?,), dtype=int64)

3. Define Feature Columns





In [7]:



    


def extend_feature_columns(feature_columns, hparams):
    
    num_buckets = hparams.num_buckets
    embedding_size = hparams.embedding_size

    buckets = np.linspace(-3, 3, num_buckets).tolist()

    alpha_X_beta = tf.feature_column.crossed_column(
            [feature_columns['alpha'], feature_columns['beta']], 4)

    x_bucketized = tf.feature_column.bucketized_column(
            feature_columns['x'], boundaries=buckets)

    y_bucketized = tf.feature_column.bucketized_column(
            feature_columns['y'], boundaries=buckets)

    x_bucketized_X_y_bucketized = tf.feature_column.crossed_column(
           [x_bucketized, y_bucketized], num_buckets**2)

    x_bucketized_X_y_bucketized_embedded = tf.feature_column.embedding_column(
            x_bucketized_X_y_bucketized, dimension=embedding_size)


    feature_columns['alpha_X_beta'] = alpha_X_beta
    feature_columns['x_bucketized_X_y_bucketized'] = x_bucketized_X_y_bucketized
    feature_columns['x_bucketized_X_y_bucketized_embedded'] = x_bucketized_X_y_bucketized_embedded
    
    return feature_columns
    

def get_feature_columns(hparams):
    
    CONSTRUCTED_NUMERIC_FEATURES_NAMES = ['x_2', 'y_2', 'xy', 'dist_xy']
    all_numeric_feature_names = NUMERIC_FEATURE_NAMES.copy() 
    
    if PROCESS_FEATURES:
        all_numeric_feature_names += CONSTRUCTED_NUMERIC_FEATURES_NAMES

    numeric_columns = {feature_name: tf.feature_column.numeric_column(feature_name)
                       for feature_name in all_numeric_feature_names}

    categorical_column_with_vocabulary = \
        {item[0]: tf.feature_column.categorical_column_with_vocabulary_list(item[0], item[1])
         for item in CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.items()}
        
    feature_columns = {}

    if numeric_columns is not None:
        feature_columns.update(numeric_columns)

    if categorical_column_with_vocabulary is not None:
        feature_columns.update(categorical_column_with_vocabulary)
    
    if EXTEND_FEATURE_COLUMNS:
        feature_columns = extend_feature_columns(feature_columns, hparams)
        
    return feature_columns

feature_columns = get_feature_columns(tf.contrib.training.HParams(num_buckets=5,embedding_size=3))
print("Feature Columns: {}".format(feature_columns))



    


















    






Feature Columns: {'x': _NumericColumn(key='x', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'y': _NumericColumn(key='y', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'x_2': _NumericColumn(key='x_2', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'y_2': _NumericColumn(key='y_2', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'xy': _NumericColumn(key='xy', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'dist_xy': _NumericColumn(key='dist_xy', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'alpha': _VocabularyListCategoricalColumn(key='alpha', vocabulary_list=('ax01', 'ax02'), dtype=tf.string, default_value=-1, num_oov_buckets=0), 'beta': _VocabularyListCategoricalColumn(key='beta', vocabulary_list=('bx01', 'bx02'), dtype=tf.string, default_value=-1, num_oov_buckets=0), 'alpha_X_beta': _CrossedColumn(keys=(_VocabularyListCategoricalColumn(key='alpha', vocabulary_list=('ax01', 'ax02'), dtype=tf.string, default_value=-1, num_oov_buckets=0), _VocabularyListCategoricalColumn(key='beta', vocabulary_list=('bx01', 'bx02'), dtype=tf.string, default_value=-1, num_oov_buckets=0)), hash_bucket_size=4, hash_key=None), 'x_bucketized_X_y_bucketized': _CrossedColumn(keys=(_BucketizedColumn(source_column=_NumericColumn(key='x', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), boundaries=(-3.0, -1.5, 0.0, 1.5, 3.0)), _BucketizedColumn(source_column=_NumericColumn(key='y', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), boundaries=(-3.0, -1.5, 0.0, 1.5, 3.0))), hash_bucket_size=25, hash_key=None), 'x_bucketized_X_y_bucketized_embedded': _EmbeddingColumn(categorical_column=_CrossedColumn(keys=(_BucketizedColumn(source_column=_NumericColumn(key='x', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), boundaries=(-3.0, -1.5, 0.0, 1.5, 3.0)), _BucketizedColumn(source_column=_NumericColumn(key='y', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), boundaries=(-3.0, -1.5, 0.0, 1.5, 3.0))), hash_bucket_size=25, hash_key=None), dimension=3, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x11fa9f908>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, max_norm=None, trainable=True)}

4. Define Model Function





In [9]:



    


def get_input_layer_feature_columns(hparams):
    
    feature_columns = list(get_feature_columns(hparams).values())
    
    dense_columns = list(
        filter(lambda column: isinstance(column, feature_column._NumericColumn) |
                              isinstance(column, feature_column._EmbeddingColumn),
               feature_columns
        )
    )

    categorical_columns = list(
        filter(lambda column: isinstance(column, feature_column._VocabularyListCategoricalColumn) |
                              isinstance(column, feature_column._BucketizedColumn),
                   feature_columns)
    )
    

    indicator_columns = list(
            map(lambda column: tf.feature_column.indicator_column(column),
                categorical_columns)
    )
    
    return dense_columns+indicator_columns

def classification_model_fn(features, labels, mode, params):

    hidden_units = params.hidden_units
    output_layer_size = len(TARGET_LABELS)

    feature_columns = get_input_layer_feature_columns(hparams)

    # Create the input layers from the feature columns
    input_layer = tf.feature_column.input_layer(features= features, 
                                                feature_columns=feature_columns)


    # Create a fully-connected layer-stack based on the hidden_units in the params
    hidden_layers = tf.contrib.layers.stack(inputs= input_layer,
                                            layer= tf.contrib.layers.fully_connected,
                                            stack_args= hidden_units)

    # Connect the output layer (logits) to the hidden layer (no activation fn)
    logits = tf.layers.dense(inputs=hidden_layers, 
                             units=output_layer_size)

    # Reshape output layer to 1-dim Tensor to return predictions
    output = tf.squeeze(logits)

    # Provide an estimator spec for `ModeKeys.PREDICT`.
    if mode == tf.estimator.ModeKeys.PREDICT:
        probabilities = tf.nn.softmax(logits)
        predicted_indices = tf.argmax(probabilities, 1)

        # Convert predicted_indices back into strings
        predictions = {
            'class': tf.gather(TARGET_LABELS, predicted_indices),
            'probabilities': probabilities
        }
        export_outputs = {
            'prediction': tf.estimator.export.PredictOutput(predictions)
        }
        
        # Provide an estimator spec for `ModeKeys.PREDICT` modes.
        return tf.estimator.EstimatorSpec(mode,
                                          predictions=predictions,
                                          export_outputs=export_outputs)

    # Calculate loss using softmax cross entropy
    loss = tf.reduce_mean(
        tf.nn.sparse_softmax_cross_entropy_with_logits(
            logits=logits, labels=labels))
    
    tf.summary.scalar('loss', loss)
    
    
    if mode == tf.estimator.ModeKeys.TRAIN:
        # Create Optimiser
        optimizer = tf.train.AdamOptimizer()

        # Create training operation
        train_op = optimizer.minimize(
            loss=loss, global_step=tf.train.get_global_step())

        # Provide an estimator spec for `ModeKeys.TRAIN` modes.
        return tf.estimator.EstimatorSpec(mode=mode,
                                          loss=loss, 
                                          train_op=train_op)
        


    if mode == tf.estimator.ModeKeys.EVAL:
        probabilities = tf.nn.softmax(logits)
        predicted_indices = tf.argmax(probabilities, 1)

        # Return accuracy and area under ROC curve metrics
        labels_one_hot = tf.one_hot(
            labels,
            depth=len(TARGET_LABELS),
            on_value=True,
            off_value=False,
            dtype=tf.bool
        )
        
        eval_metric_ops = {
            'accuracy': tf.metrics.accuracy(labels, predicted_indices),
            'auroc': tf.metrics.auc(labels_one_hot, probabilities)
        }
        
        # Provide an estimator spec for `ModeKeys.EVAL` modes.
        return tf.estimator.EstimatorSpec(mode, 
                                          loss=loss, 
                                          eval_metric_ops=eval_metric_ops)



def create_estimator(run_config, hparams):
    estimator = tf.estimator.Estimator(model_fn=classification_model_fn, 
                                  params=hparams, 
                                  config=run_config)
    
    print("")
    print("Estimator Type: {}".format(type(estimator)))
    print("")

    return estimator

6. Run Experiment

a. Define experiment function





In [10]:



    


def generate_experiment_fn(**experiment_args):

    def _experiment_fn(run_config, hparams):

        train_input_fn = lambda: csv_input_fn(
            TRAIN_DATA_FILES_PATTERN,
            mode = tf.estimator.ModeKeys.TRAIN,
            num_epochs=hparams.num_epochs,
            batch_size=hparams.batch_size
        )

        eval_input_fn = lambda: csv_input_fn(
            VALID_DATA_FILES_PATTERN,
            mode=tf.estimator.ModeKeys.EVAL,
            num_epochs=1,
            batch_size=hparams.batch_size
        )

        estimator = create_estimator(run_config, hparams)

        return tf.contrib.learn.Experiment(
            estimator,
            train_input_fn=train_input_fn,
            eval_input_fn=eval_input_fn,
            eval_steps=None,
            **experiment_args
        )

    return _experiment_fn

b. Set HParam and RunConfig





In [11]:



    


TRAIN_SIZE = 12000
NUM_EPOCHS = 1 #1000
BATCH_SIZE = 500
NUM_EVAL = 1 #10
CHECKPOINT_STEPS = int((TRAIN_SIZE/BATCH_SIZE) * (NUM_EPOCHS/NUM_EVAL))

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    hidden_units=[16, 12, 8],
    num_buckets = 6,
    embedding_size = 3,
    dropout_prob = 0.001)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.contrib.learn.RunConfig(
    save_checkpoints_steps=CHECKPOINT_STEPS,
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS)
print("Required Evaluation Steps:", NUM_EVAL) 
print("That is 1 evaluation step after each",NUM_EPOCHS/NUM_EVAL," epochs")
print("Save Checkpoint After",CHECKPOINT_STEPS,"steps")



    


















    






[('batch_size', 500), ('dropout_prob', 0.001), ('embedding_size', 3), ('hidden_units', [16, 12, 8]), ('num_buckets', 6), ('num_epochs', 1000)]
Model Directory: trained_models/class-model-02

Dataset Size: 12000
Batch Size: 500
Steps per Epoch: 24.0
Total Steps: 24000.0
Required Evaluation Steps: 10
That is 1 evaluation step after each 100.0  epochs
Save Checkpoint After 2400 steps

c. Define JSON serving function





In [29]:



    


def json_serving_input_fn():
    
    receiver_tensor = {}

    for feature_name in FEATURE_NAMES:
        dtype = tf.float32 if feature_name in NUMERIC_FEATURE_NAMES else tf.string
        receiver_tensor[feature_name] = tf.placeholder(shape=[None], dtype=dtype)

    if PROCESS_FEATURES:
        features = process_features(receiver_tensor)

    return tf.estimator.export.ServingInputReceiver(
        features, receiver_tensor)

d. Run the Experiment via learn_runner





In [13]:



    


if not RESUME_TRAINING:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...") 


tf.logging.set_verbosity(tf.logging.INFO)
 
time_start = datetime.utcnow() 
print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................") 

learn_runner.run(
    experiment_fn=generate_experiment_fn(

        export_strategies=[
            make_export_strategy(
            json_serving_input_fn,
            exports_to_keep=1,
            as_text=True
            )
        ]
    ),
    run_config=run_config,
    schedule="train_and_evaluate",
    hparams=hparams
)

time_end = datetime.utcnow() 
print(".......................................")
print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))



    


















    






Removing previous artifacts...
Experiment started at 10:01:57
.......................................
WARNING:tensorflow:RunConfig.uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.
INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11fab0668>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1
}
, '_tf_random_seed': 19830610, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': 2400, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'trained_models/class-model-02'}

Estimator Type: <class 'tensorflow.python.estimator.estimator.Estimator'>

WARNING:tensorflow:RunConfig.uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.
WARNING:tensorflow:From /Users/khalidsalama/anaconda/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:267: BaseMonitor.__init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.

* data input_fn:
================
Input file(s): data/train-*.csv
Batch size: 500
Epoch Count: 1000
Mode: train
Shuffle: True
================

INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Saving checkpoints for 1 into trained_models/class-model-02/model.ckpt.

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:02:04
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-1
INFO:tensorflow:Finished evaluation at 2017-11-16-10:02:04
INFO:tensorflow:Saving dict for global step 1: accuracy = 0.512, auroc = 0.450045, global_step = 1, loss = 0.704711
INFO:tensorflow:Validation (step 1): accuracy = 0.512, auroc = 0.450045, loss = 0.704711, global_step = 1
INFO:tensorflow:loss = 0.704995, step = 1
INFO:tensorflow:global_step/sec: 29.4915
INFO:tensorflow:loss = 0.507005, step = 101 (0.831 sec)
INFO:tensorflow:global_step/sec: 141.129
INFO:tensorflow:loss = 0.277739, step = 201 (0.710 sec)
INFO:tensorflow:global_step/sec: 136.065
INFO:tensorflow:loss = 0.177773, step = 301 (0.735 sec)
INFO:tensorflow:global_step/sec: 131.106
INFO:tensorflow:loss = 0.14348, step = 401 (0.761 sec)
INFO:tensorflow:global_step/sec: 130.596
INFO:tensorflow:loss = 0.128667, step = 501 (0.766 sec)
INFO:tensorflow:global_step/sec: 150.354
INFO:tensorflow:loss = 0.137072, step = 601 (0.665 sec)
INFO:tensorflow:global_step/sec: 149.207
INFO:tensorflow:loss = 0.115298, step = 701 (0.670 sec)
INFO:tensorflow:global_step/sec: 151.466
INFO:tensorflow:loss = 0.122777, step = 801 (0.660 sec)
INFO:tensorflow:global_step/sec: 165.756
INFO:tensorflow:loss = 0.129738, step = 901 (0.603 sec)
INFO:tensorflow:global_step/sec: 158.62
INFO:tensorflow:loss = 0.131859, step = 1001 (0.631 sec)
INFO:tensorflow:global_step/sec: 142.051
INFO:tensorflow:loss = 0.124195, step = 1101 (0.704 sec)
INFO:tensorflow:global_step/sec: 156.486
INFO:tensorflow:loss = 0.134477, step = 1201 (0.639 sec)
INFO:tensorflow:global_step/sec: 144.793
INFO:tensorflow:loss = 0.107011, step = 1301 (0.692 sec)
INFO:tensorflow:global_step/sec: 135.467
INFO:tensorflow:loss = 0.0999744, step = 1401 (0.737 sec)
INFO:tensorflow:global_step/sec: 146.207
INFO:tensorflow:loss = 0.106525, step = 1501 (0.685 sec)
INFO:tensorflow:global_step/sec: 142.292
INFO:tensorflow:loss = 0.129114, step = 1601 (0.703 sec)
INFO:tensorflow:global_step/sec: 131.858
INFO:tensorflow:loss = 0.0973879, step = 1701 (0.758 sec)
INFO:tensorflow:global_step/sec: 163.514
INFO:tensorflow:loss = 0.109529, step = 1801 (0.613 sec)
INFO:tensorflow:global_step/sec: 127.315
INFO:tensorflow:loss = 0.112641, step = 1901 (0.787 sec)
INFO:tensorflow:global_step/sec: 145.198
INFO:tensorflow:loss = 0.0889645, step = 2001 (0.687 sec)
INFO:tensorflow:global_step/sec: 153.456
INFO:tensorflow:loss = 0.147329, step = 2101 (0.652 sec)
INFO:tensorflow:global_step/sec: 122.759
INFO:tensorflow:loss = 0.107785, step = 2201 (0.821 sec)
INFO:tensorflow:global_step/sec: 131.902
INFO:tensorflow:loss = 0.11488, step = 2301 (0.752 sec)
INFO:tensorflow:Saving checkpoints for 2401 into trained_models/class-model-02/model.ckpt.
INFO:tensorflow:global_step/sec: 71.8195

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:02:24
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-2401
INFO:tensorflow:Finished evaluation at 2017-11-16-10:02:24
INFO:tensorflow:Saving dict for global step 2401: accuracy = 0.952333, auroc = 0.990932, global_step = 2401, loss = 0.12277
INFO:tensorflow:Validation (step 2401): accuracy = 0.952333, auroc = 0.990932, loss = 0.12277, global_step = 2401
INFO:tensorflow:loss = 0.0936106, step = 2401 (2.624 sec)
INFO:tensorflow:global_step/sec: 52.4154
INFO:tensorflow:loss = 0.0985235, step = 2501 (0.676 sec)
INFO:tensorflow:global_step/sec: 142.226
INFO:tensorflow:loss = 0.10412, step = 2601 (0.704 sec)
INFO:tensorflow:global_step/sec: 145.741
INFO:tensorflow:loss = 0.0910959, step = 2701 (0.686 sec)
INFO:tensorflow:global_step/sec: 144.111
INFO:tensorflow:loss = 0.123404, step = 2801 (0.694 sec)
INFO:tensorflow:global_step/sec: 143.364
INFO:tensorflow:loss = 0.0787893, step = 2901 (0.697 sec)
INFO:tensorflow:global_step/sec: 141.15
INFO:tensorflow:loss = 0.0914475, step = 3001 (0.710 sec)
INFO:tensorflow:global_step/sec: 139.22
INFO:tensorflow:loss = 0.104222, step = 3101 (0.721 sec)
INFO:tensorflow:global_step/sec: 124.501
INFO:tensorflow:loss = 0.0898928, step = 3201 (0.799 sec)
INFO:tensorflow:global_step/sec: 135.295
INFO:tensorflow:loss = 0.0869894, step = 3301 (0.741 sec)
INFO:tensorflow:global_step/sec: 134.603
INFO:tensorflow:loss = 0.0691452, step = 3401 (0.741 sec)
INFO:tensorflow:global_step/sec: 111.664
INFO:tensorflow:loss = 0.0972319, step = 3501 (0.896 sec)
INFO:tensorflow:global_step/sec: 134.069
INFO:tensorflow:loss = 0.129884, step = 3601 (0.746 sec)
INFO:tensorflow:global_step/sec: 145.324
INFO:tensorflow:loss = 0.126836, step = 3701 (0.690 sec)
INFO:tensorflow:global_step/sec: 123.757
INFO:tensorflow:loss = 0.105561, step = 3801 (0.807 sec)
INFO:tensorflow:global_step/sec: 131.251
INFO:tensorflow:loss = 0.0893482, step = 3901 (0.761 sec)
INFO:tensorflow:global_step/sec: 148.52
INFO:tensorflow:loss = 0.115303, step = 4001 (0.674 sec)
INFO:tensorflow:global_step/sec: 122.91
INFO:tensorflow:loss = 0.113887, step = 4101 (0.813 sec)
INFO:tensorflow:global_step/sec: 148.196
INFO:tensorflow:loss = 0.0996855, step = 4201 (0.675 sec)
INFO:tensorflow:global_step/sec: 121.237
INFO:tensorflow:loss = 0.112538, step = 4301 (0.826 sec)
INFO:tensorflow:global_step/sec: 133.752
INFO:tensorflow:loss = 0.0869823, step = 4401 (0.747 sec)
INFO:tensorflow:global_step/sec: 98.4703
INFO:tensorflow:loss = 0.086611, step = 4501 (1.016 sec)
INFO:tensorflow:global_step/sec: 127.052
INFO:tensorflow:loss = 0.115097, step = 4601 (0.788 sec)
INFO:tensorflow:global_step/sec: 110.141
INFO:tensorflow:loss = 0.122176, step = 4701 (0.908 sec)
INFO:tensorflow:Saving checkpoints for 4801 into trained_models/class-model-02/model.ckpt.
INFO:tensorflow:global_step/sec: 52.5798

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:02:45
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-4801
INFO:tensorflow:Finished evaluation at 2017-11-16-10:02:46
INFO:tensorflow:Saving dict for global step 4801: accuracy = 0.953, auroc = 0.990574, global_step = 4801, loss = 0.122546
INFO:tensorflow:Validation (step 4801): accuracy = 0.953, auroc = 0.990574, loss = 0.122546, global_step = 4801
INFO:tensorflow:loss = 0.0954013, step = 4801 (3.704 sec)
INFO:tensorflow:global_step/sec: 40.3061
INFO:tensorflow:loss = 0.118161, step = 4901 (0.678 sec)
INFO:tensorflow:global_step/sec: 146.684
INFO:tensorflow:loss = 0.0957444, step = 5001 (0.682 sec)
INFO:tensorflow:global_step/sec: 150.106
INFO:tensorflow:loss = 0.112879, step = 5101 (0.666 sec)
INFO:tensorflow:global_step/sec: 146.086
INFO:tensorflow:loss = 0.106377, step = 5201 (0.685 sec)
INFO:tensorflow:global_step/sec: 146.621
INFO:tensorflow:loss = 0.115252, step = 5301 (0.681 sec)
INFO:tensorflow:global_step/sec: 152.679
INFO:tensorflow:loss = 0.101358, step = 5401 (0.655 sec)
INFO:tensorflow:global_step/sec: 150.955
INFO:tensorflow:loss = 0.119894, step = 5501 (0.663 sec)
INFO:tensorflow:global_step/sec: 149.754
INFO:tensorflow:loss = 0.072343, step = 5601 (0.667 sec)
INFO:tensorflow:global_step/sec: 149.583
INFO:tensorflow:loss = 0.131565, step = 5701 (0.669 sec)
INFO:tensorflow:global_step/sec: 152.351
INFO:tensorflow:loss = 0.133214, step = 5801 (0.656 sec)
INFO:tensorflow:global_step/sec: 151.486
INFO:tensorflow:loss = 0.0863683, step = 5901 (0.660 sec)
INFO:tensorflow:global_step/sec: 144.991
INFO:tensorflow:loss = 0.091702, step = 6001 (0.692 sec)
INFO:tensorflow:global_step/sec: 109.89
INFO:tensorflow:loss = 0.104924, step = 6101 (0.914 sec)
INFO:tensorflow:global_step/sec: 86.5082
INFO:tensorflow:loss = 0.103703, step = 6201 (1.151 sec)
INFO:tensorflow:global_step/sec: 138.282
INFO:tensorflow:loss = 0.0978273, step = 6301 (0.722 sec)
INFO:tensorflow:global_step/sec: 125.078
INFO:tensorflow:loss = 0.123435, step = 6401 (0.800 sec)
INFO:tensorflow:global_step/sec: 136.906
INFO:tensorflow:loss = 0.0855783, step = 6501 (0.730 sec)
INFO:tensorflow:global_step/sec: 133.272
INFO:tensorflow:loss = 0.100026, step = 6601 (0.750 sec)
INFO:tensorflow:global_step/sec: 134.181
INFO:tensorflow:loss = 0.117686, step = 6701 (0.746 sec)
INFO:tensorflow:global_step/sec: 133.825
INFO:tensorflow:loss = 0.086353, step = 6801 (0.747 sec)
INFO:tensorflow:global_step/sec: 139.058
INFO:tensorflow:loss = 0.089118, step = 6901 (0.719 sec)
INFO:tensorflow:global_step/sec: 135.213
INFO:tensorflow:loss = 0.137246, step = 7001 (0.739 sec)
INFO:tensorflow:global_step/sec: 145.63
INFO:tensorflow:loss = 0.114853, step = 7101 (0.690 sec)
INFO:tensorflow:Saving checkpoints for 7201 into trained_models/class-model-02/model.ckpt.
INFO:tensorflow:global_step/sec: 44.0674

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:03:06
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-7201
INFO:tensorflow:Finished evaluation at 2017-11-16-10:03:06
INFO:tensorflow:Saving dict for global step 7201: accuracy = 0.953, auroc = 0.990408, global_step = 7201, loss = 0.122437
INFO:tensorflow:Validation (step 7201): accuracy = 0.953, auroc = 0.990408, loss = 0.122437, global_step = 7201
INFO:tensorflow:loss = 0.0846166, step = 7201 (4.052 sec)
INFO:tensorflow:global_step/sec: 36.0541
INFO:tensorflow:loss = 0.0932813, step = 7301 (0.990 sec)
INFO:tensorflow:global_step/sec: 98.3483
INFO:tensorflow:loss = 0.104423, step = 7401 (1.016 sec)
INFO:tensorflow:global_step/sec: 119.238
INFO:tensorflow:loss = 0.111495, step = 7501 (0.839 sec)
INFO:tensorflow:global_step/sec: 121.581
INFO:tensorflow:loss = 0.139149, step = 7601 (0.822 sec)
INFO:tensorflow:global_step/sec: 130.642
INFO:tensorflow:loss = 0.12195, step = 7701 (0.765 sec)
INFO:tensorflow:global_step/sec: 89.7305
INFO:tensorflow:loss = 0.0857602, step = 7801 (1.116 sec)
INFO:tensorflow:global_step/sec: 111.051
INFO:tensorflow:loss = 0.0914469, step = 7901 (0.900 sec)
INFO:tensorflow:global_step/sec: 134.436
INFO:tensorflow:loss = 0.0996811, step = 8001 (0.744 sec)
INFO:tensorflow:global_step/sec: 144.406
INFO:tensorflow:loss = 0.118719, step = 8101 (0.692 sec)
INFO:tensorflow:global_step/sec: 150.689
INFO:tensorflow:loss = 0.135272, step = 8201 (0.664 sec)
INFO:tensorflow:global_step/sec: 143.871
INFO:tensorflow:loss = 0.0734525, step = 8301 (0.695 sec)
INFO:tensorflow:global_step/sec: 146.193
INFO:tensorflow:loss = 0.140691, step = 8401 (0.684 sec)
INFO:tensorflow:global_step/sec: 146.277
INFO:tensorflow:loss = 0.0782113, step = 8501 (0.685 sec)
INFO:tensorflow:global_step/sec: 129.753
INFO:tensorflow:loss = 0.103149, step = 8601 (0.769 sec)
INFO:tensorflow:global_step/sec: 121.703
INFO:tensorflow:loss = 0.116948, step = 8701 (0.823 sec)
INFO:tensorflow:global_step/sec: 140.573
INFO:tensorflow:loss = 0.0981242, step = 8801 (0.711 sec)
INFO:tensorflow:global_step/sec: 112.233
INFO:tensorflow:loss = 0.134027, step = 8901 (0.892 sec)
INFO:tensorflow:global_step/sec: 137.366
INFO:tensorflow:loss = 0.0989845, step = 9001 (0.727 sec)
INFO:tensorflow:global_step/sec: 128.465
INFO:tensorflow:loss = 0.06701, step = 9101 (0.778 sec)
INFO:tensorflow:global_step/sec: 134.837
INFO:tensorflow:loss = 0.0834616, step = 9201 (0.742 sec)
INFO:tensorflow:global_step/sec: 116.897
INFO:tensorflow:loss = 0.10542, step = 9301 (0.856 sec)
INFO:tensorflow:global_step/sec: 138.951
INFO:tensorflow:loss = 0.120118, step = 9401 (0.719 sec)
INFO:tensorflow:global_step/sec: 144.881
INFO:tensorflow:loss = 0.110919, step = 9501 (0.690 sec)
INFO:tensorflow:Saving checkpoints for 9601 into trained_models/class-model-02/model.ckpt.
INFO:tensorflow:global_step/sec: 59.6816

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:03:27
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-9601
INFO:tensorflow:Finished evaluation at 2017-11-16-10:03:28
INFO:tensorflow:Saving dict for global step 9601: accuracy = 0.952667, auroc = 0.990145, global_step = 9601, loss = 0.123059
INFO:tensorflow:Validation (step 9601): accuracy = 0.952667, auroc = 0.990145, loss = 0.123059, global_step = 9601
INFO:tensorflow:loss = 0.123169, step = 9601 (3.081 sec)
INFO:tensorflow:global_step/sec: 47.9642
INFO:tensorflow:loss = 0.101235, step = 9701 (0.680 sec)
INFO:tensorflow:global_step/sec: 146.149
INFO:tensorflow:loss = 0.0856791, step = 9801 (0.683 sec)
INFO:tensorflow:global_step/sec: 144.123
INFO:tensorflow:loss = 0.117822, step = 9901 (0.694 sec)
INFO:tensorflow:global_step/sec: 129.311
INFO:tensorflow:loss = 0.114267, step = 10001 (0.774 sec)
INFO:tensorflow:global_step/sec: 122.352
INFO:tensorflow:loss = 0.0880663, step = 10101 (0.817 sec)
INFO:tensorflow:global_step/sec: 112.025
INFO:tensorflow:loss = 0.0942426, step = 10201 (0.893 sec)
INFO:tensorflow:global_step/sec: 111.021
INFO:tensorflow:loss = 0.102521, step = 10301 (0.901 sec)
INFO:tensorflow:global_step/sec: 137.512
INFO:tensorflow:loss = 0.0900848, step = 10401 (0.728 sec)
INFO:tensorflow:global_step/sec: 99.5867
INFO:tensorflow:loss = 0.0884231, step = 10501 (1.004 sec)
INFO:tensorflow:global_step/sec: 82.5767
INFO:tensorflow:loss = 0.137083, step = 10601 (1.212 sec)
INFO:tensorflow:global_step/sec: 127.347
INFO:tensorflow:loss = 0.0878391, step = 10701 (0.783 sec)
INFO:tensorflow:global_step/sec: 122.585
INFO:tensorflow:loss = 0.109161, step = 10801 (0.817 sec)
INFO:tensorflow:global_step/sec: 136.72
INFO:tensorflow:loss = 0.105165, step = 10901 (0.731 sec)
INFO:tensorflow:global_step/sec: 138.405
INFO:tensorflow:loss = 0.0764331, step = 11001 (0.722 sec)
INFO:tensorflow:global_step/sec: 149.934
INFO:tensorflow:loss = 0.102466, step = 11101 (0.667 sec)
INFO:tensorflow:global_step/sec: 144.103
INFO:tensorflow:loss = 0.0969958, step = 11201 (0.695 sec)
INFO:tensorflow:global_step/sec: 149.688
INFO:tensorflow:loss = 0.122459, step = 11301 (0.668 sec)
INFO:tensorflow:global_step/sec: 149.666
INFO:tensorflow:loss = 0.0879807, step = 11401 (0.668 sec)
INFO:tensorflow:global_step/sec: 138.23
INFO:tensorflow:loss = 0.101467, step = 11501 (0.724 sec)
INFO:tensorflow:global_step/sec: 141.647
INFO:tensorflow:loss = 0.0861694, step = 11601 (0.706 sec)
INFO:tensorflow:global_step/sec: 141.706
INFO:tensorflow:loss = 0.0751712, step = 11701 (0.705 sec)
INFO:tensorflow:global_step/sec: 141.023
INFO:tensorflow:loss = 0.134511, step = 11801 (0.709 sec)
INFO:tensorflow:global_step/sec: 148.589
INFO:tensorflow:loss = 0.091991, step = 11901 (0.673 sec)
INFO:tensorflow:Saving checkpoints for 12001 into trained_models/class-model-02/model.ckpt.
INFO:tensorflow:global_step/sec: 66.2862

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:03:48
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-12001
INFO:tensorflow:Finished evaluation at 2017-11-16-10:03:48
INFO:tensorflow:Saving dict for global step 12001: accuracy = 0.952, auroc = 0.990059, global_step = 12001, loss = 0.123726
INFO:tensorflow:Validation (step 12001): accuracy = 0.952, auroc = 0.990059, loss = 0.123726, global_step = 12001
INFO:tensorflow:loss = 0.101227, step = 12001 (2.925 sec)
INFO:tensorflow:global_step/sec: 45.8186
INFO:tensorflow:loss = 0.126308, step = 12101 (0.769 sec)
INFO:tensorflow:global_step/sec: 118.518
INFO:tensorflow:loss = 0.0658626, step = 12201 (0.842 sec)
INFO:tensorflow:global_step/sec: 117.447
INFO:tensorflow:loss = 0.0884497, step = 12301 (0.852 sec)
INFO:tensorflow:global_step/sec: 119.392
INFO:tensorflow:loss = 0.133171, step = 12401 (0.837 sec)
INFO:tensorflow:global_step/sec: 121.101
INFO:tensorflow:loss = 0.0927926, step = 12501 (0.826 sec)
INFO:tensorflow:global_step/sec: 134.509
INFO:tensorflow:loss = 0.0889409, step = 12601 (0.743 sec)
INFO:tensorflow:global_step/sec: 129.07
INFO:tensorflow:loss = 0.0953739, step = 12701 (0.775 sec)
INFO:tensorflow:global_step/sec: 127.773
INFO:tensorflow:loss = 0.0889393, step = 12801 (0.782 sec)
INFO:tensorflow:global_step/sec: 133.884
INFO:tensorflow:loss = 0.10308, step = 12901 (0.747 sec)
INFO:tensorflow:global_step/sec: 130.222
INFO:tensorflow:loss = 0.108705, step = 13001 (0.768 sec)
INFO:tensorflow:global_step/sec: 118.791
INFO:tensorflow:loss = 0.0796857, step = 13101 (0.841 sec)
INFO:tensorflow:global_step/sec: 113.503
INFO:tensorflow:loss = 0.086282, step = 13201 (0.881 sec)
INFO:tensorflow:global_step/sec: 119.457
INFO:tensorflow:loss = 0.098805, step = 13301 (0.837 sec)
INFO:tensorflow:global_step/sec: 145.32
INFO:tensorflow:loss = 0.0878216, step = 13401 (0.688 sec)
INFO:tensorflow:global_step/sec: 149.237
INFO:tensorflow:loss = 0.098558, step = 13501 (0.669 sec)
INFO:tensorflow:global_step/sec: 150.074
INFO:tensorflow:loss = 0.113125, step = 13601 (0.667 sec)
INFO:tensorflow:global_step/sec: 151.05
INFO:tensorflow:loss = 0.10725, step = 13701 (0.662 sec)
INFO:tensorflow:global_step/sec: 148.869
INFO:tensorflow:loss = 0.08377, step = 13801 (0.671 sec)
INFO:tensorflow:global_step/sec: 153.932
INFO:tensorflow:loss = 0.096439, step = 13901 (0.650 sec)
INFO:tensorflow:global_step/sec: 141.154
INFO:tensorflow:loss = 0.0940999, step = 14001 (0.709 sec)
INFO:tensorflow:global_step/sec: 143.24
INFO:tensorflow:loss = 0.110324, step = 14101 (0.699 sec)
INFO:tensorflow:global_step/sec: 141.956
INFO:tensorflow:loss = 0.117309, step = 14201 (0.704 sec)
INFO:tensorflow:global_step/sec: 108.454
INFO:tensorflow:loss = 0.0785927, step = 14301 (0.922 sec)
INFO:tensorflow:Saving checkpoints for 14401 into trained_models/class-model-02/model.ckpt.
INFO:tensorflow:global_step/sec: 54.6822

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:04:08
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-14401
INFO:tensorflow:Finished evaluation at 2017-11-16-10:04:09
INFO:tensorflow:Saving dict for global step 14401: accuracy = 0.952667, auroc = 0.990026, global_step = 14401, loss = 0.124651
INFO:tensorflow:Validation (step 14401): accuracy = 0.952667, auroc = 0.990026, loss = 0.124651, global_step = 14401
INFO:tensorflow:loss = 0.0768939, step = 14401 (3.327 sec)
INFO:tensorflow:global_step/sec: 45.6661
INFO:tensorflow:loss = 0.100866, step = 14501 (0.691 sec)
INFO:tensorflow:global_step/sec: 150.551
INFO:tensorflow:loss = 0.0970681, step = 14601 (0.664 sec)
INFO:tensorflow:global_step/sec: 158.123
INFO:tensorflow:loss = 0.096075, step = 14701 (0.633 sec)
INFO:tensorflow:global_step/sec: 154.577
INFO:tensorflow:loss = 0.119234, step = 14801 (0.647 sec)
INFO:tensorflow:global_step/sec: 161.046
INFO:tensorflow:loss = 0.107677, step = 14901 (0.621 sec)
INFO:tensorflow:global_step/sec: 164.412
INFO:tensorflow:loss = 0.073222, step = 15001 (0.608 sec)
INFO:tensorflow:global_step/sec: 153.171
INFO:tensorflow:loss = 0.12106, step = 15101 (0.654 sec)
INFO:tensorflow:global_step/sec: 153.599
INFO:tensorflow:loss = 0.0852666, step = 15201 (0.651 sec)
INFO:tensorflow:global_step/sec: 144.106
INFO:tensorflow:loss = 0.088182, step = 15301 (0.693 sec)
INFO:tensorflow:global_step/sec: 147.688
INFO:tensorflow:loss = 0.0849419, step = 15401 (0.677 sec)
INFO:tensorflow:global_step/sec: 145.633
INFO:tensorflow:loss = 0.112724, step = 15501 (0.686 sec)
INFO:tensorflow:global_step/sec: 143.636
INFO:tensorflow:loss = 0.112466, step = 15601 (0.698 sec)
INFO:tensorflow:global_step/sec: 157.8
INFO:tensorflow:loss = 0.0959474, step = 15701 (0.632 sec)
INFO:tensorflow:global_step/sec: 155.276
INFO:tensorflow:loss = 0.0914555, step = 15801 (0.644 sec)
INFO:tensorflow:global_step/sec: 156.233
INFO:tensorflow:loss = 0.0968269, step = 15901 (0.640 sec)
INFO:tensorflow:global_step/sec: 154.207
INFO:tensorflow:loss = 0.122351, step = 16001 (0.648 sec)
INFO:tensorflow:global_step/sec: 155.652
INFO:tensorflow:loss = 0.132393, step = 16101 (0.643 sec)
INFO:tensorflow:global_step/sec: 155.081
INFO:tensorflow:loss = 0.0759641, step = 16201 (0.644 sec)
INFO:tensorflow:global_step/sec: 147.507
INFO:tensorflow:loss = 0.0985603, step = 16301 (0.679 sec)
INFO:tensorflow:global_step/sec: 145.338
INFO:tensorflow:loss = 0.0996119, step = 16401 (0.688 sec)
INFO:tensorflow:global_step/sec: 143.415
INFO:tensorflow:loss = 0.107813, step = 16501 (0.698 sec)
INFO:tensorflow:global_step/sec: 135.38
INFO:tensorflow:loss = 0.0985213, step = 16601 (0.738 sec)
INFO:tensorflow:global_step/sec: 132.261
INFO:tensorflow:loss = 0.097859, step = 16701 (0.757 sec)
INFO:tensorflow:Saving checkpoints for 16801 into trained_models/class-model-02/model.ckpt.
INFO:tensorflow:global_step/sec: 66.3611

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:04:27
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-16801
INFO:tensorflow:Finished evaluation at 2017-11-16-10:04:28
INFO:tensorflow:Saving dict for global step 16801: accuracy = 0.952667, auroc = 0.990018, global_step = 16801, loss = 0.124398
INFO:tensorflow:Validation (step 16801): accuracy = 0.952667, auroc = 0.990018, loss = 0.124398, global_step = 16801
INFO:tensorflow:loss = 0.0896079, step = 16801 (3.118 sec)
INFO:tensorflow:global_step/sec: 43.6274
INFO:tensorflow:loss = 0.102983, step = 16901 (0.681 sec)
INFO:tensorflow:global_step/sec: 145.041
INFO:tensorflow:loss = 0.105697, step = 17001 (0.689 sec)
INFO:tensorflow:global_step/sec: 126.1
INFO:tensorflow:loss = 0.11775, step = 17101 (0.792 sec)
INFO:tensorflow:global_step/sec: 140.646
INFO:tensorflow:loss = 0.127795, step = 17201 (0.712 sec)
INFO:tensorflow:global_step/sec: 118.484
INFO:tensorflow:loss = 0.0882414, step = 17301 (0.844 sec)
INFO:tensorflow:global_step/sec: 105.092
INFO:tensorflow:loss = 0.0939193, step = 17401 (0.951 sec)
INFO:tensorflow:global_step/sec: 139.265
INFO:tensorflow:loss = 0.0815788, step = 17501 (0.718 sec)
INFO:tensorflow:global_step/sec: 113.715
INFO:tensorflow:loss = 0.0881263, step = 17601 (0.881 sec)
INFO:tensorflow:global_step/sec: 139.168
INFO:tensorflow:loss = 0.0965539, step = 17701 (0.717 sec)
INFO:tensorflow:global_step/sec: 128.788
INFO:tensorflow:loss = 0.128618, step = 17801 (0.778 sec)
INFO:tensorflow:global_step/sec: 134.257
INFO:tensorflow:loss = 0.105337, step = 17901 (0.744 sec)
INFO:tensorflow:global_step/sec: 141.134
INFO:tensorflow:loss = 0.0957875, step = 18001 (0.707 sec)
INFO:tensorflow:global_step/sec: 145.587
INFO:tensorflow:loss = 0.0916918, step = 18101 (0.687 sec)
INFO:tensorflow:global_step/sec: 147.475
INFO:tensorflow:loss = 0.0921837, step = 18201 (0.678 sec)
INFO:tensorflow:global_step/sec: 146.598
INFO:tensorflow:loss = 0.113692, step = 18301 (0.682 sec)
INFO:tensorflow:global_step/sec: 111.608
INFO:tensorflow:loss = 0.12167, step = 18401 (0.896 sec)
INFO:tensorflow:global_step/sec: 148.979
INFO:tensorflow:loss = 0.0867275, step = 18501 (0.672 sec)
INFO:tensorflow:global_step/sec: 144.699
INFO:tensorflow:loss = 0.0696542, step = 18601 (0.690 sec)
INFO:tensorflow:global_step/sec: 148.54
INFO:tensorflow:loss = 0.11629, step = 18701 (0.673 sec)
INFO:tensorflow:global_step/sec: 144.831
INFO:tensorflow:loss = 0.103487, step = 18801 (0.691 sec)
INFO:tensorflow:global_step/sec: 134.271
INFO:tensorflow:loss = 0.121626, step = 18901 (0.745 sec)
INFO:tensorflow:global_step/sec: 118.265
INFO:tensorflow:loss = 0.0904476, step = 19001 (0.847 sec)
INFO:tensorflow:global_step/sec: 97.4367
INFO:tensorflow:loss = 0.130144, step = 19101 (1.026 sec)
INFO:tensorflow:Saving checkpoints for 19201 into trained_models/class-model-02/model.ckpt.
INFO:tensorflow:global_step/sec: 59.7701

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:04:48
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-19201
INFO:tensorflow:Finished evaluation at 2017-11-16-10:04:48
INFO:tensorflow:Saving dict for global step 19201: accuracy = 0.952, auroc = 0.990102, global_step = 19201, loss = 0.124175
INFO:tensorflow:Validation (step 19201): accuracy = 0.952, auroc = 0.990102, loss = 0.124175, global_step = 19201
INFO:tensorflow:loss = 0.0884245, step = 19201 (3.216 sec)
INFO:tensorflow:global_step/sec: 45.3576
INFO:tensorflow:loss = 0.113087, step = 19301 (0.662 sec)
INFO:tensorflow:global_step/sec: 149.426
INFO:tensorflow:loss = 0.0974836, step = 19401 (0.669 sec)
INFO:tensorflow:global_step/sec: 150.33
INFO:tensorflow:loss = 0.095888, step = 19501 (0.665 sec)
INFO:tensorflow:global_step/sec: 150.734
INFO:tensorflow:loss = 0.113549, step = 19601 (0.664 sec)
INFO:tensorflow:global_step/sec: 151.101
INFO:tensorflow:loss = 0.118642, step = 19701 (0.662 sec)
INFO:tensorflow:global_step/sec: 147.935
INFO:tensorflow:loss = 0.109153, step = 19801 (0.677 sec)
INFO:tensorflow:global_step/sec: 141.823
INFO:tensorflow:loss = 0.0774345, step = 19901 (0.704 sec)
INFO:tensorflow:global_step/sec: 128.881
INFO:tensorflow:loss = 0.0888871, step = 20001 (0.776 sec)
INFO:tensorflow:global_step/sec: 111.231
INFO:tensorflow:loss = 0.0857756, step = 20101 (0.903 sec)
INFO:tensorflow:global_step/sec: 133.472
INFO:tensorflow:loss = 0.133486, step = 20201 (0.745 sec)
INFO:tensorflow:global_step/sec: 142.715
INFO:tensorflow:loss = 0.0899208, step = 20301 (0.700 sec)
INFO:tensorflow:global_step/sec: 147.783
INFO:tensorflow:loss = 0.0817193, step = 20401 (0.676 sec)
INFO:tensorflow:global_step/sec: 150.934
INFO:tensorflow:loss = 0.0785056, step = 20501 (0.663 sec)
INFO:tensorflow:global_step/sec: 150.671
INFO:tensorflow:loss = 0.0856874, step = 20601 (0.664 sec)
INFO:tensorflow:global_step/sec: 148.834
INFO:tensorflow:loss = 0.110973, step = 20701 (0.672 sec)
INFO:tensorflow:global_step/sec: 149.601
INFO:tensorflow:loss = 0.107857, step = 20801 (0.669 sec)
INFO:tensorflow:global_step/sec: 144.04
INFO:tensorflow:loss = 0.100954, step = 20901 (0.693 sec)
INFO:tensorflow:global_step/sec: 140.474
INFO:tensorflow:loss = 0.108373, step = 21001 (0.714 sec)
INFO:tensorflow:global_step/sec: 141.685
INFO:tensorflow:loss = 0.0970489, step = 21101 (0.705 sec)
INFO:tensorflow:global_step/sec: 141.683
INFO:tensorflow:loss = 0.0824475, step = 21201 (0.705 sec)
INFO:tensorflow:global_step/sec: 152.271
INFO:tensorflow:loss = 0.0766804, step = 21301 (0.658 sec)
INFO:tensorflow:global_step/sec: 151.085
INFO:tensorflow:loss = 0.101988, step = 21401 (0.661 sec)
INFO:tensorflow:global_step/sec: 151.445
INFO:tensorflow:loss = 0.0991419, step = 21501 (0.661 sec)
INFO:tensorflow:Saving checkpoints for 21601 into trained_models/class-model-02/model.ckpt.
INFO:tensorflow:global_step/sec: 67.0398

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:05:07
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-21601
INFO:tensorflow:Finished evaluation at 2017-11-16-10:05:07
INFO:tensorflow:Saving dict for global step 21601: accuracy = 0.953333, auroc = 0.989868, global_step = 21601, loss = 0.123933
INFO:tensorflow:Validation (step 21601): accuracy = 0.953333, auroc = 0.989868, loss = 0.123933, global_step = 21601
INFO:tensorflow:loss = 0.099648, step = 21601 (3.111 sec)
INFO:tensorflow:global_step/sec: 39.9744
INFO:tensorflow:loss = 0.111169, step = 21701 (0.884 sec)
INFO:tensorflow:global_step/sec: 111.508
INFO:tensorflow:loss = 0.0949617, step = 21801 (0.897 sec)
INFO:tensorflow:global_step/sec: 121.73
INFO:tensorflow:loss = 0.122113, step = 21901 (0.821 sec)
INFO:tensorflow:global_step/sec: 126.098
INFO:tensorflow:loss = 0.142735, step = 22001 (0.793 sec)
INFO:tensorflow:global_step/sec: 129.265
INFO:tensorflow:loss = 0.127849, step = 22101 (0.773 sec)
INFO:tensorflow:global_step/sec: 133.893
INFO:tensorflow:loss = 0.0991012, step = 22201 (0.746 sec)
INFO:tensorflow:global_step/sec: 104.953
INFO:tensorflow:loss = 0.0926553, step = 22301 (0.955 sec)
INFO:tensorflow:global_step/sec: 93.5009
INFO:tensorflow:loss = 0.109507, step = 22401 (1.069 sec)
INFO:tensorflow:global_step/sec: 86.6978
INFO:tensorflow:loss = 0.113566, step = 22501 (1.154 sec)
INFO:tensorflow:global_step/sec: 103.168
INFO:tensorflow:loss = 0.11242, step = 22601 (0.969 sec)
INFO:tensorflow:global_step/sec: 111.575
INFO:tensorflow:loss = 0.0992325, step = 22701 (0.897 sec)
INFO:tensorflow:global_step/sec: 105.787
INFO:tensorflow:loss = 0.100414, step = 22801 (0.943 sec)
INFO:tensorflow:global_step/sec: 90.4526
INFO:tensorflow:loss = 0.0970973, step = 22901 (1.106 sec)
INFO:tensorflow:global_step/sec: 141.727
INFO:tensorflow:loss = 0.0810552, step = 23001 (0.705 sec)
INFO:tensorflow:global_step/sec: 149.775
INFO:tensorflow:loss = 0.0794714, step = 23101 (0.668 sec)
INFO:tensorflow:global_step/sec: 104.964
INFO:tensorflow:loss = 0.115687, step = 23201 (0.957 sec)
INFO:tensorflow:global_step/sec: 107.01
INFO:tensorflow:loss = 0.122627, step = 23301 (0.933 sec)
INFO:tensorflow:global_step/sec: 125.658
INFO:tensorflow:loss = 0.0949917, step = 23401 (0.793 sec)
INFO:tensorflow:global_step/sec: 120.233
INFO:tensorflow:loss = 0.0853031, step = 23501 (0.832 sec)
INFO:tensorflow:global_step/sec: 123.034
INFO:tensorflow:loss = 0.0916689, step = 23601 (0.814 sec)
INFO:tensorflow:global_step/sec: 111.915
INFO:tensorflow:loss = 0.0758059, step = 23701 (0.893 sec)
INFO:tensorflow:global_step/sec: 140.874
INFO:tensorflow:loss = 0.100516, step = 23801 (0.710 sec)
INFO:tensorflow:global_step/sec: 145.384
INFO:tensorflow:loss = 0.0910458, step = 23901 (0.689 sec)
INFO:tensorflow:Saving checkpoints for 24000 into trained_models/class-model-02/model.ckpt.
INFO:tensorflow:Loss for final step: 0.0876797.

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:05:30
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-24000
INFO:tensorflow:Finished evaluation at 2017-11-16-10:05:31
INFO:tensorflow:Saving dict for global step 24000: accuracy = 0.954, auroc = 0.989835, global_step = 24000, loss = 0.124743
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-24000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: b"trained_models/class-model-02/export/Servo/temp-b'1510826732'/saved_model.pbtxt"
.......................................
Experiment finished at 10:05:32

Experiment elapsed time: 214.989123 seconds

6. Evaluate the Model





In [14]:



    


TRAIN_SIZE = 12000
VALID_SIZE = 3000
TEST_SIZE = 5000
train_input_fn = lambda: csv_input_fn(files_name_pattern= TRAIN_DATA_FILES_PATTERN, 
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= TRAIN_SIZE)

valid_input_fn = lambda: csv_input_fn(files_name_pattern= VALID_DATA_FILES_PATTERN, 
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= VALID_SIZE)

test_input_fn = lambda: csv_input_fn(files_name_pattern= TEST_DATA_FILES_PATTERN, 
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= TEST_SIZE)

estimator = create_estimator(run_config, hparams)

train_results = estimator.evaluate(input_fn=train_input_fn, steps=1)
print()
print("######################################################################################")
print("# Train Measures: {}".format(train_results))
print("######################################################################################")

valid_results = estimator.evaluate(input_fn=valid_input_fn, steps=1)
print()
print("######################################################################################")
print("# Valid Measures: {}".format(valid_results))
print("######################################################################################")

test_results = estimator.evaluate(input_fn=test_input_fn, steps=1)
print()
print("######################################################################################")
print("# Test Measures: {}".format(test_results))
print("######################################################################################")



    


















    






INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11fab0668>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1
}
, '_tf_random_seed': 19830610, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': 2400, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'trained_models/class-model-02'}

Estimator Type: <class 'tensorflow.python.estimator.estimator.Estimator'>


* data input_fn:
================
Input file(s): data/train-*.csv
Batch size: 12000
Epoch Count: None
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:05:33
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-24000
INFO:tensorflow:Evaluation [1/1]
INFO:tensorflow:Finished evaluation at 2017-11-16-10:05:34
INFO:tensorflow:Saving dict for global step 24000: accuracy = 0.9575, auroc = 0.993459, global_step = 24000, loss = 0.103367

######################################################################################
# Train Measures: {'accuracy': 0.95749998, 'auroc': 0.99345917, 'loss': 0.10336723, 'global_step': 24000}
######################################################################################

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 3000
Epoch Count: None
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:05:36
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-24000
INFO:tensorflow:Evaluation [1/1]
INFO:tensorflow:Finished evaluation at 2017-11-16-10:05:36
INFO:tensorflow:Saving dict for global step 24000: accuracy = 0.954, auroc = 0.989835, global_step = 24000, loss = 0.124743

######################################################################################
# Valid Measures: {'accuracy': 0.954, 'auroc': 0.98983496, 'loss': 0.12474335, 'global_step': 24000}
######################################################################################

* data input_fn:
================
Input file(s): data/test-*.csv
Batch size: 5000
Epoch Count: None
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:05:37
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-24000
INFO:tensorflow:Evaluation [1/1]
INFO:tensorflow:Finished evaluation at 2017-11-16-10:05:38
INFO:tensorflow:Saving dict for global step 24000: accuracy = 0.9512, auroc = 0.990328, global_step = 24000, loss = 0.125981

######################################################################################
# Test Measures: {'accuracy': 0.95120001, 'auroc': 0.99032795, 'loss': 0.12598085, 'global_step': 24000}
######################################################################################

7. Prediction





In [15]:



    


import itertools

predict_input_fn = lambda: csv_input_fn(files_name_pattern= TEST_DATA_FILES_PATTERN, 
                                      mode= tf.estimator.ModeKeys.PREDICT,
                                      batch_size= 5)

predictions = list(itertools.islice(estimator.predict(input_fn=predict_input_fn),5))

print("")

print("* Predicted Classes: {}".format(list(map(lambda item: item["class"]
    ,predictions))))

print("* Predicted Probabilities: {}".format(list(map(lambda item: list(item["probabilities"])
     ,predictions))))



    


















    






* data input_fn:
================
Input file(s): data/test-*.csv
Batch size: 5
Epoch Count: None
Mode: infer
Shuffle: False
================

WARNING:tensorflow:Input graph does not contain a QueueRunner. That means predict yields forever. This is probably a mistake.
INFO:tensorflow:Restoring parameters from trained_models/class-model-02/model.ckpt-24000

* Predicted Classes: [b'positive', b'negative', b'negative', b'positive', b'negative']
* Predicted Probabilities: [[0.9260425, 0.073957428], [4.0501654e-08, 1.0], [2.1145448e-09, 1.0], [0.99516016, 0.0048398692], [0.0096074371, 0.99039257]]

Serving Exported Model





In [16]:



    


import os

export_dir = model_dir +"/export/Servo/"

saved_model_dir = export_dir + "/" + os.listdir(path=export_dir)[-1] 

print(saved_model_dir)
print("")

predictor_fn = tf.contrib.predictor.from_saved_model(
    export_dir = saved_model_dir,
    signature_def_key="prediction"
)

output = predictor_fn(
    {
        'x': [0.5, -1],
        'y': [1, 0.5],
        'alpha': ['ax01', 'ax01'],
        'beta': ['bx02', 'bx01']
        
    }
)
print(output)



    


















    






trained_models/class-model-02/export/Servo//1510826732

INFO:tensorflow:Restoring parameters from b'trained_models/class-model-02/export/Servo//1510826732/variables/variables'
{'class': array([b'negative', b'negative'], dtype=object), 'probabilities': array([[  1.11883190e-02,   9.88811731e-01],
       [  2.19069278e-07,   9.99999762e-01]], dtype=float32)}

















In [ ]:

Content source: GoogleCloudPlatform/tf-estimator-tutorials

Similar notebooks:

notebook.community | gallery | about