notebook.community

Edit and run

Dependencies





In [1]:



    


# Tensorflow
import tensorflow as tf
print('Tested with TensorFlow 1.2.0')
print('Your TensorFlow version:', tf.__version__) 

# Feeding function for enqueue data
from tensorflow.python.estimator.inputs.queues import feeding_functions as ff

# Rnn common functions
from tensorflow.contrib.learn.python.learn.estimators import rnn_common

# Model builder
from tensorflow.python.estimator import model_fn as model_fn_lib

# Run an experiment
from tensorflow.contrib.learn.python.learn import learn_runner

# Helpers for data processing
import pandas as pd
import numpy as np
import argparse
import random



    


















    






Tested with TensorFLow 1.2.0
Your TensorFlow version: 1.2.0

Loading Data

First, we want to create our word vectors. For simplicity, we're going to be using a pretrained model.

As one of the biggest players in the ML game, Google was able to train a Word2Vec model on a massive Google News dataset that contained over 100 billion different words! From that model, Google was able to create 3 million word vectors, each with a dimensionality of 300.

In an ideal scenario, we'd use those vectors, but since the word vectors matrix is quite large (3.6 GB!), we'll be using a much more manageable matrix that is trained using GloVe, a similar word vector generation model. The matrix will contain 400,000 word vectors, each with a dimensionality of 50.

We're going to be importing two different data structures, one will be a Python list with the 400,000 words, and one will be a 400,000 x 50 dimensional embedding matrix that holds all of the word vector values.





In [2]:



    


# data from: http://ai.stanford.edu/~amaas/data/sentiment/
TRAIN_INPUT = 'data/train.csv'
TEST_INPUT = 'data/test.csv'

# data manually generated
MY_TEST_INPUT = 'data/mytest.csv'

# wordtovec
# https://nlp.stanford.edu/projects/glove/
# the matrix will contain 400,000 word vectors, each with a dimensionality of 50.
word_list = np.load('word_list.npy')
word_list = word_list.tolist() # originally loaded as numpy array
word_list = [word.decode('UTF-8') for word in word_list] # encode words as UTF-8
print('Loaded the word list, length:', len(word_list))

word_vector = np.load('word_vector.npy')
print ('Loaded the word vector, shape:', word_vector.shape)



    


















    






Loaded the word list, length: 400000
Loaded the word vector, shape: (400000, 50)

We can search our word list for a word like "baseball", and then access its corresponding vector through the embedding matrix.





In [3]:



    


baseball_index = word_list.index('baseball')
print('Example: baseball')
print(word_vector[baseball_index])



    


















    






Example: baseball
[-1.93270004  1.04209995 -0.78514999  0.91033     0.22711    -0.62158
 -1.64929998  0.07686    -0.58679998  0.058831    0.35628     0.68915999
 -0.50598001  0.70472997  1.26639998 -0.40031001 -0.020687    0.80862999
 -0.90565997 -0.074054   -0.87674999 -0.62910002 -0.12684999  0.11524
 -0.55685002 -1.68260002 -0.26291001  0.22632     0.713      -1.08280003
  2.12310004  0.49869001  0.066711   -0.48225999 -0.17896999  0.47699001
  0.16384     0.16537    -0.11506    -0.15962    -0.94926    -0.42833
 -0.59456998  1.35660005 -0.27506     0.19918001 -0.36008     0.55667001
 -0.70314997  0.17157   ]

Now that we have our vectors, our first step is taking an input sentence and then constructing the its vector representation. Let's say that we have the input sentence "I thought the movie was incredible and inspiring". In order to get the word vectors, we can use Tensorflow's embedding lookup function. This function takes in two arguments, one for the embedding matrix (the wordVectors matrix in our case), and one for the ids of each of the words. The ids vector can be thought of as the integerized representation of the training set. This is basically just the row index of each of the words. Let's look at a quick example to make this concrete.





In [92]:



    


max_seq_length = 10 # maximum length of sentence
num_dims = 50 # dimensions for each word vector

first_sentence = np.zeros((max_seq_length), dtype='int32')
first_sentence[0] = word_list.index("i")
first_sentence[1] = word_list.index("thought")
first_sentence[2] = word_list.index("the")
first_sentence[3] = word_list.index("movie")
first_sentence[4] = word_list.index("was")
first_sentence[5] = word_list.index("incredible")
first_sentence[6] = word_list.index("and")
first_sentence[7] = word_list.index("inspiring")
# first_sentence[8] = 0
# first_sentence[9] = 0

print(first_sentence.shape)
print(first_sentence) # shows the row index for each word



    


















    






(10,)
[    41    804 201534   1005     15   7446      5  13767      0      0]

TODO### Insert image

The 10 x 50 output should contain the 50 dimensional word vectors for each of the 10 words in the sequence.





In [5]:



    


with tf.Session() as sess:
    print(tf.nn.embedding_lookup(word_vector, first_sentence).eval().shape)



    


















    






(10, 50)

Before creating the ids matrix for the whole training set, let’s first take some time to visualize the type of data that we have. This will help us determine the best value for setting our maximum sequence length. In the previous example, we used a max length of 10, but this value is largely dependent on the inputs you have.

The training set we're going to use is the Imdb movie review dataset. This set has 25,000 movie reviews, with 12,500 positive reviews and 12,500 negative reviews. Each of the reviews is stored in a txt file that we need to parse through. The positive reviews are stored in one directory and the negative reviews are stored in another. The following piece of code will determine total and average number of words in each review.





In [6]:



    


from os import listdir
from os.path import isfile, join
positiveFiles = ['positiveReviews/' + f for f in listdir('positiveReviews/') if isfile(join('positiveReviews/', f))]
negativeFiles = ['negativeReviews/' + f for f in listdir('negativeReviews/') if isfile(join('negativeReviews/', f))]
numWords = []
for pf in positiveFiles:
    with open(pf, "r", encoding='utf-8') as f:
        line=f.readline()
        counter = len(line.split())
        numWords.append(counter)       
print('Positive files finished')

for nf in negativeFiles:
    with open(nf, "r", encoding='utf-8') as f:
        line=f.readline()
        counter = len(line.split())
        numWords.append(counter)  
print('Negative files finished')

numFiles = len(numWords)
print('The total number of files is', numFiles)
print('The total number of words in the files is', sum(numWords))
print('The average number of words in the files is', sum(numWords)/len(numWords))



    


















    






Positive files finished
Negative files finished
The total number of files is 25000
The total number of words in the files is 5844680
The average number of words in the files is 233.7872

We can also use the Matplot library to visualize this data in a histogram format.





In [7]:



    


import matplotlib.pyplot as plt
%matplotlib inline
plt.hist(numWords, 50)
plt.xlabel('Sequence Length')
plt.ylabel('Frequency')
plt.axis([0, 1200, 0, 8000])
plt.show()

From the histogram as well as the average number of words per file, we can safely say that most reviews will fall under 250 words, which is the max sequence length value we will set.





In [8]:



    


max_seq_len = 250

Data





In [73]:



    


ids_matrix = np.load('ids_matrix.npy').tolist()

Parameters





In [80]:



    


# Parameters for training
STEPS = 15000
BATCH_SIZE = 32

# Parameters for data processing
REVIEW_KEY = 'review'
SEQUENCE_LENGTH_KEY = 'sequence_length'

Separating train and test data

The training set we're going to use is the Imdb movie review dataset. This set has 25,000 movie reviews, with 12,500 positive reviews and 12,500 negative reviews.

Let's first give a positive label [1, 0] to the first 12500 reviews, and a negative label [0, 1] to the other reviews.





In [75]:



    


POSITIVE_REVIEWS = 12500

# copying sequences
data_sequences = [np.asarray(v, dtype=np.int32) for v in ids_matrix]
# generating labels
data_labels = [[1, 0] if i < POSITIVE_REVIEWS else [0, 1] for i in range(len(ids_matrix))]
# also creating a length column, this will be used by the Dynamic RNN
# see more about it here: https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn
data_length = [max_seq_len for i in range(len(ids_matrix))]

Then, let's shuffle the data and use 90% of the reviews for training and the other 10% for testing.





In [76]:



    


data = list(zip(data_sequences, data_labels, data_length))
random.shuffle(data) # shuffle

data = np.asarray(data)
# separating train and test data
limit = int(len(data) * 0.9)

train_data = data[:limit]
test_data = data[limit:]

Verifying if the train and test data have enough positive and negative examples





In [77]:



    


LABEL_INDEX = 1
def _number_of_pos_labels(df):
    pos_labels = 0
    for value in df:
        if value[LABEL_INDEX] == [1, 0]:
            pos_labels += 1
    return pos_labels

pos_labels_train = _number_of_pos_labels(train_data)
total_labels_train = len(train_data)

pos_labels_test = _number_of_pos_labels(test_data)
total_labels_test = len(test_data)

print('Total number of positive labels:', pos_labels_train + pos_labels_test)
print('Proportion of positive labels on the Train data:', pos_labels_train/total_labels_train)
print('Proportion of positive labels on the Test data:', pos_labels_test/total_labels_test)



    


















    






Total number of positive labels: 12500
Proportion of positive labels on the Train data: 0.49933333333333335
Proportion of positive labels on the Test data: 0.506

Input functions





In [159]:



    


def get_input_fn(df, batch_size, num_epochs=1, shuffle=True):  
    def input_fn():
        
        sequences = np.asarray([v for v in df[:,0]], dtype=np.int32)
        labels = np.asarray([v for v in df[:,1]], dtype=np.int32)
        length = np.asarray(df[:,2], dtype=np.int32)

        # https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/data
        dataset = (
            tf.contrib.data.Dataset.from_tensor_slices((sequences, labels, length)) # reading data from memory
            .repeat(num_epochs) # repeat dataset the number of epochs
            .batch(batch_size)
        )
        
        # for our "manual" test we don't want to shuffle the data
        if shuffle:
            dataset = dataset.shuffle(buffer_size=100000)

        # create iterator
        review, label, length = dataset.make_one_shot_iterator().get_next()

        features = {
            REVIEW_KEY: review,
            SEQUENCE_LENGTH_KEY: length,
        }

        return features, label
    return input_fn



    











In [160]:



    


features, label = get_input_fn(test_data, 2, shuffle=False)()



    











In [161]:



    


with tf.Session() as sess:
    items = sess.run(features)
    print(items[REVIEW_KEY])

    print(sess.run(label))



    


















    






[[    36     29   7503    978    465     10 201534    371     65     34
     102      7  12474      6 201534    621      4    567    264   2500
  201534   1607    153      3 201534    281     14     12     20   2444
       4    480   1003     19   2532   6769     65      5     14   1096
      34    301      5    266     12  21853     44    973      4 201534
    2500      3   1487   7763    439     73     37    102     14    182
     749    164      6      7 399999 106337   3349 399999   1301  99048
   11027     13      7   4706 399999 399999      7    333   1983    151
  201534   1570      3  59651     32  12734 201534    371     19   4424
     142   1670   1222    152    164 399999    992   9742    197    109
     246     86     39    234 201534 399999    635      3 201534  22866
    2913      6 201534 399999  33830  24445   2115 201534    215   8183
     295   2956 217684      4    359 399999    401   4537   2280     46
      36      5     76     34     36    338     65     56    941   1088
     615     73     81     94   6597      7   4403 399999 399999 399999
      41    303      4    253   6494    142    161      5 399999   6412
   12193     41     54   1716   8273  14789      0      0      0      0
       0      0      0      0      0      0      0      0      0      0
       0      0      0      0      0      0      0      0      0      0
       0      0      0      0      0      0      0      0      0      0
       0      0      0      0      0      0      0      0      0      0
       0      0      0      0      0      0      0      0      0      0
       0      0      0      0      0      0      0      0      0      0
       0      0      0      0      0      0      0      0      0      0
       0      0      0      0      0      0      0      0      0      0]
 [    37   1005     14     36 399999     58     48    591 399999     40
      12  32795   1945      4    159     20   1089 399999     48     31
  399999   4014 399999   4333     46 399999 399999    588      7   5988
  399999      3 201534   5059 399999    273 399999     84   2182      4
    2199 399999     94    591     30      7 399999     10      7   1594
       3  16580      5      7   1594      3 399999   2050     14     36
     191 399999 399999    169    285    551     13 201534 399999     10
  201534   2661 399999 399999 399999  12073     20     94     33     51
  399999     41 399999    253 399999    127 399999     14    440     73
    1645    599 399999     41   5020 399999      7    219 399999     20
    1349      7    530   1078   1896     37     14 201534    611   1062
       3 201534 214247    238     20     10      7    191   5115 399999
      20    149     36   1089 399999     81    303 201534  34357     56
    5320 399999     66 201534   7118   1255 399999   1062      3 204834
    1176 399999  24235   1666      7    365   3747  91549     25    285
  399999     81   1716     37 399999     43    965     30      7  15002
       5   7894 399999     81     32      7    567 399999    414    303
       4   2065     60   7118 399999 204834 399999   5300   9492     87
    5976   1720   3910 201534     58     87   2459  34357      5     17
     557   1410      0      0      0      0      0      0      0      0
       0      0      0      0      0      0      0      0      0      0
       0      0      0      0      0      0      0      0      0      0
       0      0      0      0      0      0      0      0      0      0
       0      0      0      0      0      0      0      0      0      0
       0      0      0      0      0      0      0      0      0      0]]
[[0 1]
 [0 1]]

















In [83]:



    


train_input_fn = get_input_fn(train_data, BATCH_SIZE, None)
test_input_fn = get_input_fn(test_data, BATCH_SIZE)

Creating the Estimator model





In [166]:



    


def get_model_fn(rnn_cell_sizes,
                 label_dimension,
                 dnn_layer_sizes=[],
                 optimizer='SGD',
                 learning_rate=0.01,
                 embed_dim=128):
    
    def model_fn(features, labels, mode):
        
        review = features[REVIEW_KEY]
        sequence_length = tf.cast(features[SEQUENCE_LENGTH_KEY], tf.int32)

        # Creating embedding
        data = tf.Variable(tf.zeros([BATCH_SIZE, max_seq_len, 50]),dtype=tf.float32)
        data = tf.nn.embedding_lookup(word_vector, review)
        
        # Each RNN layer will consist of a LSTM cell
        rnn_layers = [tf.nn.rnn_cell.LSTMCell(size) for size in rnn_cell_sizes]
        
        # Construct the layers
        multi_rnn_cell = tf.nn.rnn_cell.MultiRNNCell(rnn_layers)
        
        # Runs the RNN model dynamically
        # more about it at: 
        # https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn
        outputs, final_state = tf.nn.dynamic_rnn(cell=multi_rnn_cell,
                                                 inputs=data,
                                                 dtype=tf.float32)

        # Slice to keep only the last cell of the RNN
        last_activations = rnn_common.select_last_activations(outputs, sequence_length)

        # Construct dense layers on top of the last cell of the RNN
        for units in dnn_layer_sizes:
            last_activations = tf.layers.dense(
              last_activations, units, activation=tf.nn.relu)
        
        # Final dense layer for prediction
        predictions = tf.layers.dense(last_activations, label_dimension)
        predictions_softmax = tf.nn.softmax(predictions)
        
        loss = None
        train_op = None
        eval_op = None
        
        preds_op = {
            'prediction': predictions_softmax,
            'label': labels
        }
        
        if mode == tf.estimator.ModeKeys.EVAL:
            eval_op = {
                "accuracy": tf.metrics.accuracy(
                         tf.argmax(input=predictions_softmax, axis=1),
                         tf.argmax(input=labels, axis=1))
            }
        
        if mode != tf.estimator.ModeKeys.PREDICT:    
            loss = tf.losses.softmax_cross_entropy(labels, predictions)
    
        if mode == tf.estimator.ModeKeys.TRAIN:    
            train_op = tf.contrib.layers.optimize_loss(
              loss,
              tf.contrib.framework.get_global_step(),
              optimizer=optimizer,
              learning_rate=learning_rate)
        
        return model_fn_lib.EstimatorSpec(mode,
                                          predictions=predictions_softmax,
                                          loss=loss,
                                          train_op=train_op,
                                          eval_metric_ops=eval_op)
    return model_fn



    











In [167]:



    


model_fn = get_model_fn(rnn_cell_sizes=[64], # size of the hidden layers
                        label_dimension=2, # since are just 2 classes
                        dnn_layer_sizes=[128, 64], # size of units in the dense layers on top of the RNN
                        optimizer='Adam',
                        learning_rate=0.001,
                        embed_dim=512)

Create and Run Experiment





In [90]:



    


# create experiment
def generate_experiment_fn():
  
    """
        Create an experiment function given hyperparameters.
        Returns:
        A function (output_dir) -> Experiment where output_dir is a string
        representing the location of summaries, checkpoints, and exports.
        this function is used by learn_runner to create an Experiment which
        executes model code provided in the form of an Estimator and
        input functions.
        All listed arguments in the outer function are used to create an
        Estimator, and input functions (training, evaluation, serving).
        Unlisted args are passed through to Experiment.
    """

    def _experiment_fn(run_config, hparams):
        estimator = tf.estimator.Estimator(model_fn=model_fn, config=run_config)
        return tf.contrib.learn.Experiment(
            estimator,
            train_input_fn=train_input_fn,
            eval_input_fn=test_input_fn,
            train_steps=STEPS
        )
    return _experiment_fn



    











In [91]:



    


# run experiment 
learn_runner.run(generate_experiment_fn(), run_config=tf.contrib.learn.RunConfig(model_dir='testing2'))



    


















    






WARNING:tensorflow:RunConfig.uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.
INFO:tensorflow:Using config: {'_save_checkpoints_steps': None, '_session_config': None, '_keep_checkpoint_every_n_hours': 10000, '_save_summary_steps': 100, '_num_ps_replicas': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f3a18f5a550>, '_tf_random_seed': None, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_model_dir': 'testing2', '_task_id': 0, '_evaluation_master': '', '_environment': 'local', '_master': '', '_is_chief': True, '_save_checkpoints_secs': 600, '_task_type': None, '_keep_checkpoint_max': 5, '_num_worker_replicas': 0}
WARNING:tensorflow:RunConfig.uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.
WARNING:tensorflow:From /usr/local/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:268: BaseMonitor.__init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
INFO:tensorflow:Create CheckpointSaverHook.










    






/usr/local/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "










    






INFO:tensorflow:Saving checkpoints for 1 into testing2/model.ckpt.
INFO:tensorflow:loss = 0.702224, step = 1
INFO:tensorflow:Starting evaluation at 2017-06-29-13:09:09
INFO:tensorflow:Restoring parameters from testing2/model.ckpt-1
INFO:tensorflow:Evaluation [1/100]
INFO:tensorflow:Evaluation [2/100]
INFO:tensorflow:Evaluation [3/100]
INFO:tensorflow:Evaluation [4/100]
INFO:tensorflow:Evaluation [5/100]
INFO:tensorflow:Evaluation [6/100]
INFO:tensorflow:Evaluation [7/100]
INFO:tensorflow:Evaluation [8/100]
INFO:tensorflow:Evaluation [9/100]
INFO:tensorflow:Evaluation [10/100]
INFO:tensorflow:Evaluation [11/100]
INFO:tensorflow:Evaluation [12/100]
INFO:tensorflow:Evaluation [13/100]
INFO:tensorflow:Evaluation [14/100]
INFO:tensorflow:Evaluation [15/100]
INFO:tensorflow:Evaluation [16/100]
INFO:tensorflow:Evaluation [17/100]
INFO:tensorflow:Evaluation [18/100]
INFO:tensorflow:Evaluation [19/100]
INFO:tensorflow:Evaluation [20/100]
INFO:tensorflow:Evaluation [21/100]
INFO:tensorflow:Evaluation [22/100]
INFO:tensorflow:Evaluation [23/100]
INFO:tensorflow:Evaluation [24/100]
INFO:tensorflow:Evaluation [25/100]
INFO:tensorflow:Evaluation [26/100]
INFO:tensorflow:Evaluation [27/100]
INFO:tensorflow:Evaluation [28/100]
INFO:tensorflow:Evaluation [29/100]
INFO:tensorflow:Evaluation [30/100]
INFO:tensorflow:Evaluation [31/100]
INFO:tensorflow:Evaluation [32/100]
INFO:tensorflow:Evaluation [33/100]
INFO:tensorflow:Evaluation [34/100]
INFO:tensorflow:Evaluation [35/100]
INFO:tensorflow:Evaluation [36/100]
INFO:tensorflow:Evaluation [37/100]
INFO:tensorflow:Evaluation [38/100]
INFO:tensorflow:Evaluation [39/100]
INFO:tensorflow:Evaluation [40/100]
INFO:tensorflow:Evaluation [41/100]
INFO:tensorflow:Evaluation [42/100]
INFO:tensorflow:Evaluation [43/100]
INFO:tensorflow:Evaluation [44/100]
INFO:tensorflow:Evaluation [45/100]
INFO:tensorflow:Evaluation [46/100]
INFO:tensorflow:Evaluation [47/100]
INFO:tensorflow:Evaluation [48/100]
INFO:tensorflow:Evaluation [49/100]
INFO:tensorflow:Evaluation [50/100]
INFO:tensorflow:Evaluation [51/100]
INFO:tensorflow:Evaluation [52/100]
INFO:tensorflow:Evaluation [53/100]
INFO:tensorflow:Evaluation [54/100]
INFO:tensorflow:Evaluation [55/100]
INFO:tensorflow:Evaluation [56/100]
INFO:tensorflow:Evaluation [57/100]
INFO:tensorflow:Evaluation [58/100]
INFO:tensorflow:Evaluation [59/100]
INFO:tensorflow:Evaluation [60/100]
INFO:tensorflow:Evaluation [61/100]
INFO:tensorflow:Evaluation [62/100]
INFO:tensorflow:Evaluation [63/100]
INFO:tensorflow:Evaluation [64/100]
INFO:tensorflow:Evaluation [65/100]
INFO:tensorflow:Evaluation [66/100]
INFO:tensorflow:Evaluation [67/100]
INFO:tensorflow:Evaluation [68/100]
INFO:tensorflow:Evaluation [69/100]
INFO:tensorflow:Evaluation [70/100]
INFO:tensorflow:Evaluation [71/100]
INFO:tensorflow:Evaluation [72/100]
INFO:tensorflow:Evaluation [73/100]
INFO:tensorflow:Evaluation [74/100]
INFO:tensorflow:Evaluation [75/100]
INFO:tensorflow:Evaluation [76/100]
INFO:tensorflow:Evaluation [77/100]
INFO:tensorflow:Evaluation [78/100]
INFO:tensorflow:Evaluation [79/100]
INFO:tensorflow:Finished evaluation at 2017-06-29-13:09:14
INFO:tensorflow:Saving dict for global step 1: accuracy = 0.5084, global_step = 1, loss = 0.692447
INFO:tensorflow:Validation (step 1): accuracy = 0.5084, global_step = 1, loss = 0.692447
INFO:tensorflow:global_step/sec: 5.77426
INFO:tensorflow:loss = 0.709747, step = 101 (17.319 sec)
INFO:tensorflow:global_step/sec: 10.8153
INFO:tensorflow:loss = 0.663808, step = 201 (9.246 sec)
INFO:tensorflow:global_step/sec: 10.9954
INFO:tensorflow:loss = 0.679195, step = 301 (9.095 sec)
INFO:tensorflow:global_step/sec: 10.8191
INFO:tensorflow:loss = 0.688457, step = 401 (9.243 sec)
INFO:tensorflow:global_step/sec: 11.059
INFO:tensorflow:loss = 0.646141, step = 501 (9.042 sec)
INFO:tensorflow:global_step/sec: 11.0316
INFO:tensorflow:loss = 0.662266, step = 601 (9.065 sec)
INFO:tensorflow:global_step/sec: 10.9618
INFO:tensorflow:loss = 0.689469, step = 701 (9.123 sec)
INFO:tensorflow:global_step/sec: 10.7993
INFO:tensorflow:loss = 0.702996, step = 801 (9.260 sec)
INFO:tensorflow:global_step/sec: 10.6876
INFO:tensorflow:loss = 0.702799, step = 901 (9.357 sec)
INFO:tensorflow:global_step/sec: 10.8427
INFO:tensorflow:loss = 0.694457, step = 1001 (9.223 sec)
INFO:tensorflow:global_step/sec: 10.653
INFO:tensorflow:loss = 0.698411, step = 1101 (9.387 sec)
INFO:tensorflow:global_step/sec: 10.4024
INFO:tensorflow:loss = 0.681778, step = 1201 (9.613 sec)
INFO:tensorflow:global_step/sec: 10.6156
INFO:tensorflow:loss = 0.6818, step = 1301 (9.420 sec)
INFO:tensorflow:global_step/sec: 11.0213
INFO:tensorflow:loss = 0.690526, step = 1401 (9.073 sec)
INFO:tensorflow:global_step/sec: 10.9186
INFO:tensorflow:loss = 0.689741, step = 1501 (9.159 sec)
INFO:tensorflow:global_step/sec: 10.7999
INFO:tensorflow:loss = 0.69076, step = 1601 (9.259 sec)
INFO:tensorflow:global_step/sec: 10.9234
INFO:tensorflow:loss = 0.695023, step = 1701 (9.155 sec)
INFO:tensorflow:global_step/sec: 10.8265
INFO:tensorflow:loss = 0.693486, step = 1801 (9.237 sec)
INFO:tensorflow:global_step/sec: 10.9657
INFO:tensorflow:loss = 0.694407, step = 1901 (9.119 sec)
INFO:tensorflow:global_step/sec: 10.9904
INFO:tensorflow:loss = 0.70909, step = 2001 (9.099 sec)
INFO:tensorflow:global_step/sec: 10.9457
INFO:tensorflow:loss = 0.69318, step = 2101 (9.136 sec)
INFO:tensorflow:global_step/sec: 10.9819
INFO:tensorflow:loss = 0.69311, step = 2201 (9.106 sec)
INFO:tensorflow:global_step/sec: 10.0748
INFO:tensorflow:loss = 0.691624, step = 2301 (9.926 sec)
INFO:tensorflow:global_step/sec: 9.88195
INFO:tensorflow:loss = 0.696891, step = 2401 (10.119 sec)
INFO:tensorflow:global_step/sec: 10.3211
INFO:tensorflow:loss = 0.694153, step = 2501 (9.689 sec)
INFO:tensorflow:global_step/sec: 10.4255
INFO:tensorflow:loss = 0.693779, step = 2601 (9.592 sec)
INFO:tensorflow:global_step/sec: 10.4922
INFO:tensorflow:loss = 0.686663, step = 2701 (9.531 sec)
INFO:tensorflow:global_step/sec: 10.3552
INFO:tensorflow:loss = 0.689399, step = 2801 (9.657 sec)
INFO:tensorflow:global_step/sec: 10.4535
INFO:tensorflow:loss = 0.692903, step = 2901 (9.566 sec)
INFO:tensorflow:global_step/sec: 10.6268
INFO:tensorflow:loss = 0.657506, step = 3001 (9.410 sec)
INFO:tensorflow:global_step/sec: 10.3449
INFO:tensorflow:loss = 0.688102, step = 3101 (9.667 sec)
INFO:tensorflow:global_step/sec: 10.308
INFO:tensorflow:loss = 0.69259, step = 3201 (9.701 sec)
INFO:tensorflow:global_step/sec: 10.1049
INFO:tensorflow:loss = 0.694794, step = 3301 (9.896 sec)
INFO:tensorflow:global_step/sec: 10.2761
INFO:tensorflow:loss = 0.687033, step = 3401 (9.731 sec)
INFO:tensorflow:global_step/sec: 10.7702
INFO:tensorflow:loss = 0.692886, step = 3501 (9.285 sec)
INFO:tensorflow:global_step/sec: 10.6976
INFO:tensorflow:loss = 0.693455, step = 3601 (9.348 sec)
INFO:tensorflow:global_step/sec: 10.7679
INFO:tensorflow:loss = 0.694858, step = 3701 (9.287 sec)
INFO:tensorflow:global_step/sec: 10.6501
INFO:tensorflow:loss = 0.689595, step = 3801 (9.390 sec)
INFO:tensorflow:global_step/sec: 10.6292
INFO:tensorflow:loss = 0.749494, step = 3901 (9.408 sec)
INFO:tensorflow:global_step/sec: 9.97154
INFO:tensorflow:loss = 0.67323, step = 4001 (10.029 sec)
INFO:tensorflow:global_step/sec: 10.4696
INFO:tensorflow:loss = 0.704927, step = 4101 (9.552 sec)
INFO:tensorflow:global_step/sec: 10.4357
INFO:tensorflow:loss = 0.664953, step = 4201 (9.582 sec)
INFO:tensorflow:global_step/sec: 10.5034
INFO:tensorflow:loss = 0.698912, step = 4301 (9.521 sec)
INFO:tensorflow:global_step/sec: 10.4302
INFO:tensorflow:loss = 0.693565, step = 4401 (9.587 sec)
INFO:tensorflow:global_step/sec: 10.3919
INFO:tensorflow:loss = 0.694478, step = 4501 (9.623 sec)
INFO:tensorflow:global_step/sec: 10.4539
INFO:tensorflow:loss = 0.680164, step = 4601 (9.566 sec)
INFO:tensorflow:global_step/sec: 10.3179
INFO:tensorflow:loss = 0.696179, step = 4701 (9.692 sec)
INFO:tensorflow:global_step/sec: 10.6305
INFO:tensorflow:loss = 0.693791, step = 4801 (9.407 sec)
INFO:tensorflow:global_step/sec: 10.5707
INFO:tensorflow:loss = 0.691543, step = 4901 (9.460 sec)
INFO:tensorflow:global_step/sec: 10.9516
INFO:tensorflow:loss = 0.687856, step = 5001 (9.131 sec)
INFO:tensorflow:global_step/sec: 10.859
INFO:tensorflow:loss = 0.659731, step = 5101 (9.209 sec)
INFO:tensorflow:global_step/sec: 10.95
INFO:tensorflow:loss = 0.751493, step = 5201 (9.132 sec)
INFO:tensorflow:global_step/sec: 10.8978
INFO:tensorflow:loss = 0.652734, step = 5301 (9.176 sec)
INFO:tensorflow:global_step/sec: 10.6529
INFO:tensorflow:loss = 0.701441, step = 5401 (9.387 sec)
INFO:tensorflow:global_step/sec: 10.6547
INFO:tensorflow:loss = 0.693071, step = 5501 (9.386 sec)
INFO:tensorflow:global_step/sec: 10.549
INFO:tensorflow:loss = 0.692545, step = 5601 (9.480 sec)
INFO:tensorflow:global_step/sec: 10.8424
INFO:tensorflow:loss = 0.680459, step = 5701 (9.223 sec)
INFO:tensorflow:global_step/sec: 10.0042
INFO:tensorflow:loss = 0.693177, step = 5801 (9.996 sec)
INFO:tensorflow:global_step/sec: 10.6846
INFO:tensorflow:loss = 0.696385, step = 5901 (9.359 sec)
INFO:tensorflow:global_step/sec: 10.6851
INFO:tensorflow:loss = 0.694919, step = 6001 (9.359 sec)
INFO:tensorflow:global_step/sec: 10.7049
INFO:tensorflow:loss = 0.680568, step = 6101 (9.342 sec)
INFO:tensorflow:global_step/sec: 10.616
INFO:tensorflow:loss = 0.725044, step = 6201 (9.420 sec)
INFO:tensorflow:Saving checkpoints for 6282 into testing2/model.ckpt.
INFO:tensorflow:Starting evaluation at 2017-06-29-13:19:09
INFO:tensorflow:Restoring parameters from testing2/model.ckpt-6282
INFO:tensorflow:Evaluation [1/100]
INFO:tensorflow:Evaluation [2/100]
INFO:tensorflow:Evaluation [3/100]
INFO:tensorflow:Evaluation [4/100]
INFO:tensorflow:Evaluation [5/100]
INFO:tensorflow:Evaluation [6/100]
INFO:tensorflow:Evaluation [7/100]
INFO:tensorflow:Evaluation [8/100]
INFO:tensorflow:Evaluation [9/100]
INFO:tensorflow:Evaluation [10/100]
INFO:tensorflow:Evaluation [11/100]
INFO:tensorflow:Evaluation [12/100]
INFO:tensorflow:Evaluation [13/100]
INFO:tensorflow:Evaluation [14/100]
INFO:tensorflow:Evaluation [15/100]
INFO:tensorflow:Evaluation [16/100]
INFO:tensorflow:Evaluation [17/100]
INFO:tensorflow:Evaluation [18/100]
INFO:tensorflow:Evaluation [19/100]
INFO:tensorflow:Evaluation [20/100]
INFO:tensorflow:Evaluation [21/100]
INFO:tensorflow:Evaluation [22/100]
INFO:tensorflow:Evaluation [23/100]
INFO:tensorflow:Evaluation [24/100]
INFO:tensorflow:Evaluation [25/100]
INFO:tensorflow:Evaluation [26/100]
INFO:tensorflow:Evaluation [27/100]
INFO:tensorflow:Evaluation [28/100]
INFO:tensorflow:Evaluation [29/100]
INFO:tensorflow:Evaluation [30/100]
INFO:tensorflow:Evaluation [31/100]
INFO:tensorflow:Evaluation [32/100]
INFO:tensorflow:Evaluation [33/100]
INFO:tensorflow:Evaluation [34/100]
INFO:tensorflow:Evaluation [35/100]
INFO:tensorflow:Evaluation [36/100]
INFO:tensorflow:Evaluation [37/100]
INFO:tensorflow:Evaluation [38/100]
INFO:tensorflow:Evaluation [39/100]
INFO:tensorflow:Evaluation [40/100]
INFO:tensorflow:Evaluation [41/100]
INFO:tensorflow:Evaluation [42/100]
INFO:tensorflow:Evaluation [43/100]
INFO:tensorflow:Evaluation [44/100]
INFO:tensorflow:Evaluation [45/100]
INFO:tensorflow:Evaluation [46/100]
INFO:tensorflow:Evaluation [47/100]
INFO:tensorflow:Evaluation [48/100]
INFO:tensorflow:Evaluation [49/100]
INFO:tensorflow:Evaluation [50/100]
INFO:tensorflow:Evaluation [51/100]
INFO:tensorflow:Evaluation [52/100]
INFO:tensorflow:Evaluation [53/100]
INFO:tensorflow:Evaluation [54/100]
INFO:tensorflow:Evaluation [55/100]
INFO:tensorflow:Evaluation [56/100]
INFO:tensorflow:Evaluation [57/100]
INFO:tensorflow:Evaluation [58/100]
INFO:tensorflow:Evaluation [59/100]
INFO:tensorflow:Evaluation [60/100]
INFO:tensorflow:Evaluation [61/100]
INFO:tensorflow:Evaluation [62/100]
INFO:tensorflow:Evaluation [63/100]
INFO:tensorflow:Evaluation [64/100]
INFO:tensorflow:Evaluation [65/100]
INFO:tensorflow:Evaluation [66/100]
INFO:tensorflow:Evaluation [67/100]
INFO:tensorflow:Evaluation [68/100]
INFO:tensorflow:Evaluation [69/100]
INFO:tensorflow:Evaluation [70/100]
INFO:tensorflow:Evaluation [71/100]
INFO:tensorflow:Evaluation [72/100]
INFO:tensorflow:Evaluation [73/100]
INFO:tensorflow:Evaluation [74/100]
INFO:tensorflow:Evaluation [75/100]
INFO:tensorflow:Evaluation [76/100]
INFO:tensorflow:Evaluation [77/100]
INFO:tensorflow:Evaluation [78/100]
INFO:tensorflow:Evaluation [79/100]
INFO:tensorflow:Finished evaluation at 2017-06-29-13:19:13
INFO:tensorflow:Saving dict for global step 6282: accuracy = 0.5644, global_step = 6282, loss = 0.672052
INFO:tensorflow:Validation (step 6282): accuracy = 0.5644, global_step = 6282, loss = 0.672052
INFO:tensorflow:global_step/sec: 6.91504
INFO:tensorflow:loss = 0.680763, step = 6301 (14.461 sec)
INFO:tensorflow:global_step/sec: 10.2158
INFO:tensorflow:loss = 0.683204, step = 6401 (9.789 sec)
INFO:tensorflow:global_step/sec: 10.4113
INFO:tensorflow:loss = 0.641561, step = 6501 (9.605 sec)
INFO:tensorflow:global_step/sec: 10.513
INFO:tensorflow:loss = 0.715619, step = 6601 (9.512 sec)
INFO:tensorflow:global_step/sec: 10.6212
INFO:tensorflow:loss = 0.677847, step = 6701 (9.415 sec)
INFO:tensorflow:global_step/sec: 10.5124
INFO:tensorflow:loss = 0.588227, step = 6801 (9.513 sec)
INFO:tensorflow:global_step/sec: 10.6684
INFO:tensorflow:loss = 0.694563, step = 6901 (9.373 sec)
INFO:tensorflow:global_step/sec: 10.4209
INFO:tensorflow:loss = 0.689794, step = 7001 (9.596 sec)
INFO:tensorflow:global_step/sec: 10.6475
INFO:tensorflow:loss = 0.685898, step = 7101 (9.392 sec)
INFO:tensorflow:global_step/sec: 10.6997
INFO:tensorflow:loss = 0.68545, step = 7201 (9.346 sec)
INFO:tensorflow:global_step/sec: 9.95594
INFO:tensorflow:loss = 0.648137, step = 7301 (10.044 sec)
INFO:tensorflow:global_step/sec: 10.2897
INFO:tensorflow:loss = 0.531919, step = 7401 (9.718 sec)
INFO:tensorflow:global_step/sec: 10.6325
INFO:tensorflow:loss = 0.662547, step = 7501 (9.405 sec)
INFO:tensorflow:global_step/sec: 10.4012
INFO:tensorflow:loss = 0.519547, step = 7601 (9.614 sec)
INFO:tensorflow:global_step/sec: 10.9943
INFO:tensorflow:loss = 0.385066, step = 7701 (9.096 sec)
INFO:tensorflow:global_step/sec: 10.8404
INFO:tensorflow:loss = 0.439061, step = 7801 (9.225 sec)
INFO:tensorflow:global_step/sec: 10.3278
INFO:tensorflow:loss = 0.220641, step = 7901 (9.683 sec)
INFO:tensorflow:global_step/sec: 9.60152
INFO:tensorflow:loss = 0.526734, step = 8001 (10.415 sec)
INFO:tensorflow:global_step/sec: 10.9489
INFO:tensorflow:loss = 0.458283, step = 8101 (9.133 sec)
INFO:tensorflow:global_step/sec: 10.5164
INFO:tensorflow:loss = 0.426452, step = 8201 (9.509 sec)
INFO:tensorflow:global_step/sec: 10.5491
INFO:tensorflow:loss = 0.406577, step = 8301 (9.479 sec)
INFO:tensorflow:global_step/sec: 11.0236
INFO:tensorflow:loss = 0.586833, step = 8401 (9.073 sec)
INFO:tensorflow:global_step/sec: 10.954
INFO:tensorflow:loss = 0.382276, step = 8501 (9.127 sec)
INFO:tensorflow:global_step/sec: 10.3982
INFO:tensorflow:loss = 0.253388, step = 8601 (9.617 sec)
INFO:tensorflow:global_step/sec: 10.5247
INFO:tensorflow:loss = 0.56352, step = 8701 (9.501 sec)
INFO:tensorflow:global_step/sec: 10.4862
INFO:tensorflow:loss = 0.453695, step = 8801 (9.536 sec)
INFO:tensorflow:global_step/sec: 10.4503
INFO:tensorflow:loss = 0.592724, step = 8901 (9.569 sec)
INFO:tensorflow:global_step/sec: 10.5971
INFO:tensorflow:loss = 0.478521, step = 9001 (9.437 sec)
INFO:tensorflow:global_step/sec: 10.9681
INFO:tensorflow:loss = 0.498691, step = 9101 (9.117 sec)
INFO:tensorflow:global_step/sec: 10.7513
INFO:tensorflow:loss = 0.475168, step = 9201 (9.301 sec)
INFO:tensorflow:global_step/sec: 10.446
INFO:tensorflow:loss = 0.332189, step = 9301 (9.573 sec)
INFO:tensorflow:global_step/sec: 10.2459
INFO:tensorflow:loss = 0.411877, step = 9401 (9.760 sec)
INFO:tensorflow:global_step/sec: 10.6161
INFO:tensorflow:loss = 0.219225, step = 9501 (9.420 sec)
INFO:tensorflow:global_step/sec: 9.92121
INFO:tensorflow:loss = 0.359576, step = 9601 (10.080 sec)
INFO:tensorflow:global_step/sec: 10.4171
INFO:tensorflow:loss = 0.338412, step = 9701 (9.600 sec)
INFO:tensorflow:global_step/sec: 10.5304
INFO:tensorflow:loss = 0.418522, step = 9801 (9.496 sec)
INFO:tensorflow:global_step/sec: 10.4868
INFO:tensorflow:loss = 0.396225, step = 9901 (9.536 sec)
INFO:tensorflow:global_step/sec: 10.2108
INFO:tensorflow:loss = 0.353239, step = 10001 (9.794 sec)
INFO:tensorflow:global_step/sec: 10.4707
INFO:tensorflow:loss = 0.316909, step = 10101 (9.550 sec)
INFO:tensorflow:global_step/sec: 10.5245
INFO:tensorflow:loss = 0.251563, step = 10201 (9.502 sec)
INFO:tensorflow:global_step/sec: 10.4461
INFO:tensorflow:loss = 0.344501, step = 10301 (9.573 sec)
INFO:tensorflow:global_step/sec: 10.522
INFO:tensorflow:loss = 0.372052, step = 10401 (9.504 sec)
INFO:tensorflow:global_step/sec: 10.5911
INFO:tensorflow:loss = 0.319963, step = 10501 (9.442 sec)
INFO:tensorflow:global_step/sec: 10.5216
INFO:tensorflow:loss = 0.369244, step = 10601 (9.504 sec)
INFO:tensorflow:global_step/sec: 10.521
INFO:tensorflow:loss = 0.264259, step = 10701 (9.505 sec)
INFO:tensorflow:global_step/sec: 10.5213
INFO:tensorflow:loss = 0.443012, step = 10801 (9.505 sec)
INFO:tensorflow:global_step/sec: 10.522
INFO:tensorflow:loss = 0.328671, step = 10901 (9.504 sec)
INFO:tensorflow:global_step/sec: 10.4649
INFO:tensorflow:loss = 0.370794, step = 11001 (9.556 sec)
INFO:tensorflow:global_step/sec: 10.521
INFO:tensorflow:loss = 0.274471, step = 11101 (9.505 sec)
INFO:tensorflow:global_step/sec: 10.3697
INFO:tensorflow:loss = 0.361685, step = 11201 (9.643 sec)
INFO:tensorflow:global_step/sec: 10.4652
INFO:tensorflow:loss = 0.291704, step = 11301 (9.555 sec)
INFO:tensorflow:global_step/sec: 10.5341
INFO:tensorflow:loss = 0.273949, step = 11401 (9.493 sec)
INFO:tensorflow:global_step/sec: 10.4882
INFO:tensorflow:loss = 0.247468, step = 11501 (9.535 sec)
INFO:tensorflow:global_step/sec: 10.5441
INFO:tensorflow:loss = 0.245574, step = 11601 (9.484 sec)
INFO:tensorflow:global_step/sec: 10.5348
INFO:tensorflow:loss = 0.632971, step = 11701 (9.492 sec)
INFO:tensorflow:global_step/sec: 10.4848
INFO:tensorflow:loss = 0.411189, step = 11801 (9.538 sec)
INFO:tensorflow:global_step/sec: 10.5086
INFO:tensorflow:loss = 0.355353, step = 11901 (9.516 sec)
INFO:tensorflow:global_step/sec: 10.5229
INFO:tensorflow:loss = 0.252377, step = 12001 (9.503 sec)
INFO:tensorflow:global_step/sec: 10.598
INFO:tensorflow:loss = 0.352639, step = 12101 (9.436 sec)
INFO:tensorflow:global_step/sec: 10.7476
INFO:tensorflow:loss = 0.292668, step = 12201 (9.304 sec)
INFO:tensorflow:global_step/sec: 10.8645
INFO:tensorflow:loss = 0.115168, step = 12301 (9.204 sec)
INFO:tensorflow:global_step/sec: 10.9346
INFO:tensorflow:loss = 0.394358, step = 12401 (9.145 sec)
INFO:tensorflow:global_step/sec: 10.9483
INFO:tensorflow:loss = 0.417252, step = 12501 (9.134 sec)
INFO:tensorflow:Saving checkpoints for 12547 into testing2/model.ckpt.
INFO:tensorflow:Starting evaluation at 2017-06-29-13:29:08
INFO:tensorflow:Restoring parameters from testing2/model.ckpt-12547
INFO:tensorflow:Evaluation [1/100]
INFO:tensorflow:Evaluation [2/100]
INFO:tensorflow:Evaluation [3/100]
INFO:tensorflow:Evaluation [4/100]
INFO:tensorflow:Evaluation [5/100]
INFO:tensorflow:Evaluation [6/100]
INFO:tensorflow:Evaluation [7/100]
INFO:tensorflow:Evaluation [8/100]
INFO:tensorflow:Evaluation [9/100]
INFO:tensorflow:Evaluation [10/100]
INFO:tensorflow:Evaluation [11/100]
INFO:tensorflow:Evaluation [12/100]
INFO:tensorflow:Evaluation [13/100]
INFO:tensorflow:Evaluation [14/100]
INFO:tensorflow:Evaluation [15/100]
INFO:tensorflow:Evaluation [16/100]
INFO:tensorflow:Evaluation [17/100]
INFO:tensorflow:Evaluation [18/100]
INFO:tensorflow:Evaluation [19/100]
INFO:tensorflow:Evaluation [20/100]
INFO:tensorflow:Evaluation [21/100]
INFO:tensorflow:Evaluation [22/100]
INFO:tensorflow:Evaluation [23/100]
INFO:tensorflow:Evaluation [24/100]
INFO:tensorflow:Evaluation [25/100]
INFO:tensorflow:Evaluation [26/100]
INFO:tensorflow:Evaluation [27/100]
INFO:tensorflow:Evaluation [28/100]
INFO:tensorflow:Evaluation [29/100]
INFO:tensorflow:Evaluation [30/100]
INFO:tensorflow:Evaluation [31/100]
INFO:tensorflow:Evaluation [32/100]
INFO:tensorflow:Evaluation [33/100]
INFO:tensorflow:Evaluation [34/100]
INFO:tensorflow:Evaluation [35/100]
INFO:tensorflow:Evaluation [36/100]
INFO:tensorflow:Evaluation [37/100]
INFO:tensorflow:Evaluation [38/100]
INFO:tensorflow:Evaluation [39/100]
INFO:tensorflow:Evaluation [40/100]
INFO:tensorflow:Evaluation [41/100]
INFO:tensorflow:Evaluation [42/100]
INFO:tensorflow:Evaluation [43/100]
INFO:tensorflow:Evaluation [44/100]
INFO:tensorflow:Evaluation [45/100]
INFO:tensorflow:Evaluation [46/100]
INFO:tensorflow:Evaluation [47/100]
INFO:tensorflow:Evaluation [48/100]
INFO:tensorflow:Evaluation [49/100]
INFO:tensorflow:Evaluation [50/100]
INFO:tensorflow:Evaluation [51/100]
INFO:tensorflow:Evaluation [52/100]
INFO:tensorflow:Evaluation [53/100]
INFO:tensorflow:Evaluation [54/100]
INFO:tensorflow:Evaluation [55/100]
INFO:tensorflow:Evaluation [56/100]
INFO:tensorflow:Evaluation [57/100]
INFO:tensorflow:Evaluation [58/100]
INFO:tensorflow:Evaluation [59/100]
INFO:tensorflow:Evaluation [60/100]
INFO:tensorflow:Evaluation [61/100]
INFO:tensorflow:Evaluation [62/100]
INFO:tensorflow:Evaluation [63/100]
INFO:tensorflow:Evaluation [64/100]
INFO:tensorflow:Evaluation [65/100]
INFO:tensorflow:Evaluation [66/100]
INFO:tensorflow:Evaluation [67/100]
INFO:tensorflow:Evaluation [68/100]
INFO:tensorflow:Evaluation [69/100]
INFO:tensorflow:Evaluation [70/100]
INFO:tensorflow:Evaluation [71/100]
INFO:tensorflow:Evaluation [72/100]
INFO:tensorflow:Evaluation [73/100]
INFO:tensorflow:Evaluation [74/100]
INFO:tensorflow:Evaluation [75/100]
INFO:tensorflow:Evaluation [76/100]
INFO:tensorflow:Evaluation [77/100]
INFO:tensorflow:Evaluation [78/100]
INFO:tensorflow:Evaluation [79/100]
INFO:tensorflow:Finished evaluation at 2017-06-29-13:29:11
INFO:tensorflow:Saving dict for global step 12547: accuracy = 0.84, global_step = 12547, loss = 0.357392
INFO:tensorflow:Validation (step 12547): accuracy = 0.84, global_step = 12547, loss = 0.357392
INFO:tensorflow:global_step/sec: 7.79841
INFO:tensorflow:loss = 0.168538, step = 12601 (12.823 sec)
INFO:tensorflow:global_step/sec: 10.8611
INFO:tensorflow:loss = 0.190768, step = 12701 (9.207 sec)
INFO:tensorflow:global_step/sec: 10.7893
INFO:tensorflow:loss = 0.377361, step = 12801 (9.268 sec)
INFO:tensorflow:global_step/sec: 10.8096
INFO:tensorflow:loss = 0.169103, step = 12901 (9.251 sec)
INFO:tensorflow:global_step/sec: 10.7698
INFO:tensorflow:loss = 0.249736, step = 13001 (9.285 sec)
INFO:tensorflow:global_step/sec: 10.888
INFO:tensorflow:loss = 0.374019, step = 13101 (9.184 sec)
INFO:tensorflow:global_step/sec: 10.942
INFO:tensorflow:loss = 0.578929, step = 13201 (9.139 sec)
INFO:tensorflow:global_step/sec: 11.0056
INFO:tensorflow:loss = 0.425376, step = 13301 (9.086 sec)
INFO:tensorflow:global_step/sec: 10.8779
INFO:tensorflow:loss = 0.384443, step = 13401 (9.193 sec)
INFO:tensorflow:global_step/sec: 10.8776
INFO:tensorflow:loss = 0.322913, step = 13501 (9.193 sec)
INFO:tensorflow:global_step/sec: 10.9114
INFO:tensorflow:loss = 0.385036, step = 13601 (9.165 sec)
INFO:tensorflow:global_step/sec: 10.8789
INFO:tensorflow:loss = 0.140187, step = 13701 (9.192 sec)
INFO:tensorflow:global_step/sec: 10.9905
INFO:tensorflow:loss = 0.287861, step = 13801 (9.099 sec)
INFO:tensorflow:global_step/sec: 10.9527
INFO:tensorflow:loss = 0.362631, step = 13901 (9.130 sec)
INFO:tensorflow:global_step/sec: 10.9066
INFO:tensorflow:loss = 0.509308, step = 14001 (9.169 sec)
INFO:tensorflow:global_step/sec: 10.9685
INFO:tensorflow:loss = 0.240151, step = 14101 (9.117 sec)
INFO:tensorflow:global_step/sec: 10.9161
INFO:tensorflow:loss = 0.205992, step = 14201 (9.161 sec)
INFO:tensorflow:global_step/sec: 10.9554
INFO:tensorflow:loss = 0.217344, step = 14301 (9.128 sec)
INFO:tensorflow:global_step/sec: 10.7673
INFO:tensorflow:loss = 0.230715, step = 14401 (9.287 sec)
INFO:tensorflow:global_step/sec: 10.8307
INFO:tensorflow:loss = 0.276864, step = 14501 (9.233 sec)
INFO:tensorflow:global_step/sec: 10.7863
INFO:tensorflow:loss = 0.263115, step = 14601 (9.271 sec)
INFO:tensorflow:global_step/sec: 10.9323
INFO:tensorflow:loss = 0.281702, step = 14701 (9.147 sec)
INFO:tensorflow:global_step/sec: 10.8883
INFO:tensorflow:loss = 0.199386, step = 14801 (9.184 sec)
INFO:tensorflow:global_step/sec: 10.8822
INFO:tensorflow:loss = 0.116956, step = 14901 (9.189 sec)
INFO:tensorflow:global_step/sec: 10.9012
INFO:tensorflow:loss = 0.0649144, step = 15001 (9.173 sec)
INFO:tensorflow:global_step/sec: 10.8143
INFO:tensorflow:loss = 0.335126, step = 15101 (9.247 sec)
INFO:tensorflow:global_step/sec: 10.8845
INFO:tensorflow:loss = 0.386393, step = 15201 (9.187 sec)
INFO:tensorflow:global_step/sec: 10.8657
INFO:tensorflow:loss = 0.20567, step = 15301 (9.203 sec)
INFO:tensorflow:global_step/sec: 10.5443
INFO:tensorflow:loss = 0.198978, step = 15401 (9.484 sec)
INFO:tensorflow:global_step/sec: 10.5136
INFO:tensorflow:loss = 0.230477, step = 15501 (9.512 sec)
INFO:tensorflow:global_step/sec: 10.5629
INFO:tensorflow:loss = 0.208272, step = 15601 (9.467 sec)
INFO:tensorflow:global_step/sec: 10.6166
INFO:tensorflow:loss = 0.303766, step = 15701 (9.419 sec)
INFO:tensorflow:global_step/sec: 10.5211
INFO:tensorflow:loss = 0.398745, step = 15801 (9.504 sec)
INFO:tensorflow:global_step/sec: 10.5106
INFO:tensorflow:loss = 0.190603, step = 15901 (9.514 sec)
INFO:tensorflow:global_step/sec: 10.4468
INFO:tensorflow:loss = 0.415988, step = 16001 (9.572 sec)
INFO:tensorflow:global_step/sec: 10.5555
INFO:tensorflow:loss = 0.424059, step = 16101 (9.474 sec)
INFO:tensorflow:global_step/sec: 10.0741
INFO:tensorflow:loss = 0.253785, step = 16201 (9.926 sec)
INFO:tensorflow:global_step/sec: 10.4485
INFO:tensorflow:loss = 0.112803, step = 16301 (9.571 sec)
INFO:tensorflow:global_step/sec: 10.1263
INFO:tensorflow:loss = 0.25306, step = 16401 (9.875 sec)
INFO:tensorflow:global_step/sec: 10.4117
INFO:tensorflow:loss = 0.144182, step = 16501 (9.605 sec)
INFO:tensorflow:global_step/sec: 10.1748
INFO:tensorflow:loss = 0.281317, step = 16601 (9.828 sec)
INFO:tensorflow:global_step/sec: 10.2507
INFO:tensorflow:loss = 0.0880013, step = 16701 (9.755 sec)
INFO:tensorflow:global_step/sec: 10.0706
INFO:tensorflow:loss = 0.175008, step = 16801 (9.930 sec)
INFO:tensorflow:global_step/sec: 9.91204
INFO:tensorflow:loss = 0.216902, step = 16901 (10.089 sec)
INFO:tensorflow:global_step/sec: 9.93574
INFO:tensorflow:loss = 0.393399, step = 17001 (10.065 sec)
INFO:tensorflow:global_step/sec: 10.3329
INFO:tensorflow:loss = 0.294201, step = 17101 (9.678 sec)
INFO:tensorflow:global_step/sec: 10.0167
INFO:tensorflow:loss = 0.237406, step = 17201 (9.983 sec)
INFO:tensorflow:global_step/sec: 10.2399
INFO:tensorflow:loss = 0.266638, step = 17301 (9.766 sec)
INFO:tensorflow:global_step/sec: 10.3119
INFO:tensorflow:loss = 0.282131, step = 17401 (9.698 sec)
INFO:tensorflow:global_step/sec: 10.0057
INFO:tensorflow:loss = 0.334409, step = 17501 (9.994 sec)
INFO:tensorflow:global_step/sec: 10.3245
INFO:tensorflow:loss = 0.402662, step = 17601 (9.686 sec)
INFO:tensorflow:global_step/sec: 10.4525
INFO:tensorflow:loss = 0.109814, step = 17701 (9.567 sec)
INFO:tensorflow:global_step/sec: 10.2608
INFO:tensorflow:loss = 0.206623, step = 17801 (9.746 sec)
INFO:tensorflow:global_step/sec: 10.5391
INFO:tensorflow:loss = 0.172913, step = 17901 (9.488 sec)
INFO:tensorflow:global_step/sec: 10.5477
INFO:tensorflow:loss = 0.106871, step = 18001 (9.481 sec)
INFO:tensorflow:global_step/sec: 10.3579
INFO:tensorflow:loss = 0.215493, step = 18101 (9.655 sec)
INFO:tensorflow:global_step/sec: 10.1419
INFO:tensorflow:loss = 0.195881, step = 18201 (9.860 sec)
INFO:tensorflow:global_step/sec: 10.7757
INFO:tensorflow:loss = 0.163682, step = 18301 (9.280 sec)
INFO:tensorflow:global_step/sec: 10.3338
INFO:tensorflow:loss = 0.138341, step = 18401 (9.677 sec)
INFO:tensorflow:global_step/sec: 10.2334
INFO:tensorflow:loss = 0.204241, step = 18501 (9.772 sec)
INFO:tensorflow:global_step/sec: 10.5684
INFO:tensorflow:loss = 0.117975, step = 18601 (9.462 sec)
INFO:tensorflow:global_step/sec: 10.6854
INFO:tensorflow:loss = 0.157836, step = 18701 (9.359 sec)
INFO:tensorflow:global_step/sec: 10.7622
INFO:tensorflow:loss = 0.0623179, step = 18801 (9.292 sec)
INFO:tensorflow:Saving checkpoints for 18858 into testing2/model.ckpt.
INFO:tensorflow:Starting evaluation at 2017-06-29-13:39:08
INFO:tensorflow:Restoring parameters from testing2/model.ckpt-18858
INFO:tensorflow:Evaluation [1/100]
INFO:tensorflow:Evaluation [2/100]
INFO:tensorflow:Evaluation [3/100]
INFO:tensorflow:Evaluation [4/100]
INFO:tensorflow:Evaluation [5/100]
INFO:tensorflow:Evaluation [6/100]
INFO:tensorflow:Evaluation [7/100]
INFO:tensorflow:Evaluation [8/100]
INFO:tensorflow:Evaluation [9/100]
INFO:tensorflow:Evaluation [10/100]
INFO:tensorflow:Evaluation [11/100]
INFO:tensorflow:Evaluation [12/100]
INFO:tensorflow:Evaluation [13/100]
INFO:tensorflow:Evaluation [14/100]
INFO:tensorflow:Evaluation [15/100]
INFO:tensorflow:Evaluation [16/100]
INFO:tensorflow:Evaluation [17/100]
INFO:tensorflow:Evaluation [18/100]
INFO:tensorflow:Evaluation [19/100]
INFO:tensorflow:Evaluation [20/100]
INFO:tensorflow:Evaluation [21/100]
INFO:tensorflow:Evaluation [22/100]
INFO:tensorflow:Evaluation [23/100]
INFO:tensorflow:Evaluation [24/100]
INFO:tensorflow:Evaluation [25/100]
INFO:tensorflow:Evaluation [26/100]
INFO:tensorflow:Evaluation [27/100]
INFO:tensorflow:Evaluation [28/100]
INFO:tensorflow:Evaluation [29/100]
INFO:tensorflow:Evaluation [30/100]
INFO:tensorflow:Evaluation [31/100]
INFO:tensorflow:Evaluation [32/100]
INFO:tensorflow:Evaluation [33/100]
INFO:tensorflow:Evaluation [34/100]
INFO:tensorflow:Evaluation [35/100]
INFO:tensorflow:Evaluation [36/100]
INFO:tensorflow:Evaluation [37/100]
INFO:tensorflow:Evaluation [38/100]
INFO:tensorflow:Evaluation [39/100]
INFO:tensorflow:Evaluation [40/100]
INFO:tensorflow:Evaluation [41/100]
INFO:tensorflow:Evaluation [42/100]
INFO:tensorflow:Evaluation [43/100]
INFO:tensorflow:Evaluation [44/100]
INFO:tensorflow:Evaluation [45/100]
INFO:tensorflow:Evaluation [46/100]
INFO:tensorflow:Evaluation [47/100]
INFO:tensorflow:Evaluation [48/100]
INFO:tensorflow:Evaluation [49/100]
INFO:tensorflow:Evaluation [50/100]
INFO:tensorflow:Evaluation [51/100]
INFO:tensorflow:Evaluation [52/100]
INFO:tensorflow:Evaluation [53/100]
INFO:tensorflow:Evaluation [54/100]
INFO:tensorflow:Evaluation [55/100]
INFO:tensorflow:Evaluation [56/100]
INFO:tensorflow:Evaluation [57/100]
INFO:tensorflow:Evaluation [58/100]
INFO:tensorflow:Evaluation [59/100]
INFO:tensorflow:Evaluation [60/100]
INFO:tensorflow:Evaluation [61/100]
INFO:tensorflow:Evaluation [62/100]
INFO:tensorflow:Evaluation [63/100]
INFO:tensorflow:Evaluation [64/100]
INFO:tensorflow:Evaluation [65/100]
INFO:tensorflow:Evaluation [66/100]
INFO:tensorflow:Evaluation [67/100]
INFO:tensorflow:Evaluation [68/100]
INFO:tensorflow:Evaluation [69/100]
INFO:tensorflow:Evaluation [70/100]
INFO:tensorflow:Evaluation [71/100]
INFO:tensorflow:Evaluation [72/100]
INFO:tensorflow:Evaluation [73/100]
INFO:tensorflow:Evaluation [74/100]
INFO:tensorflow:Evaluation [75/100]
INFO:tensorflow:Evaluation [76/100]
INFO:tensorflow:Evaluation [77/100]
INFO:tensorflow:Evaluation [78/100]
INFO:tensorflow:Evaluation [79/100]
INFO:tensorflow:Finished evaluation at 2017-06-29-13:39:12
INFO:tensorflow:Saving dict for global step 18858: accuracy = 0.8496, global_step = 18858, loss = 0.393123
INFO:tensorflow:Validation (step 18858): accuracy = 0.8496, global_step = 18858, loss = 0.393123
INFO:tensorflow:global_step/sec: 7.63898
INFO:tensorflow:loss = 0.296969, step = 18901 (13.091 sec)
INFO:tensorflow:global_step/sec: 10.134
INFO:tensorflow:loss = 0.198794, step = 19001 (9.869 sec)
INFO:tensorflow:global_step/sec: 9.90147
INFO:tensorflow:loss = 0.355534, step = 19101 (10.098 sec)
INFO:tensorflow:global_step/sec: 10.1805
INFO:tensorflow:loss = 0.231664, step = 19201 (9.823 sec)
INFO:tensorflow:global_step/sec: 10.1964
INFO:tensorflow:loss = 0.233768, step = 19301 (9.807 sec)
INFO:tensorflow:global_step/sec: 10.1765
INFO:tensorflow:loss = 0.121018, step = 19401 (9.827 sec)
INFO:tensorflow:global_step/sec: 10.4554
INFO:tensorflow:loss = 0.0577645, step = 19501 (9.565 sec)
INFO:tensorflow:global_step/sec: 10.2567
INFO:tensorflow:loss = 0.267758, step = 19601 (9.750 sec)
INFO:tensorflow:global_step/sec: 9.81198
INFO:tensorflow:loss = 0.305589, step = 19701 (10.192 sec)
INFO:tensorflow:global_step/sec: 10.0452
INFO:tensorflow:loss = 0.129044, step = 19801 (9.955 sec)
INFO:tensorflow:global_step/sec: 9.61078
INFO:tensorflow:loss = 0.0934473, step = 19901 (10.405 sec)
INFO:tensorflow:global_step/sec: 9.84472
INFO:tensorflow:loss = 0.0573027, step = 20001 (10.158 sec)
INFO:tensorflow:global_step/sec: 10.0328
INFO:tensorflow:loss = 0.14195, step = 20101 (9.967 sec)
INFO:tensorflow:global_step/sec: 9.67057
INFO:tensorflow:loss = 0.262287, step = 20201 (10.341 sec)
INFO:tensorflow:global_step/sec: 9.54714
INFO:tensorflow:loss = 0.102364, step = 20301 (10.474 sec)
INFO:tensorflow:global_step/sec: 10.0709
INFO:tensorflow:loss = 0.0309285, step = 20401 (9.930 sec)
INFO:tensorflow:global_step/sec: 10.0926
INFO:tensorflow:loss = 0.0468883, step = 20501 (9.908 sec)
INFO:tensorflow:global_step/sec: 10.056
INFO:tensorflow:loss = 0.0746669, step = 20601 (9.944 sec)
INFO:tensorflow:global_step/sec: 10.1143
INFO:tensorflow:loss = 0.412788, step = 20701 (9.887 sec)
INFO:tensorflow:global_step/sec: 9.8962
INFO:tensorflow:loss = 0.228101, step = 20801 (10.105 sec)
INFO:tensorflow:global_step/sec: 10.3426
INFO:tensorflow:loss = 0.184479, step = 20901 (9.669 sec)
INFO:tensorflow:global_step/sec: 9.95871
INFO:tensorflow:loss = 0.105869, step = 21001 (10.041 sec)
INFO:tensorflow:global_step/sec: 10.2425
INFO:tensorflow:loss = 0.0978621, step = 21101 (9.763 sec)
INFO:tensorflow:global_step/sec: 10.2865
INFO:tensorflow:loss = 0.050883, step = 21201 (9.721 sec)
INFO:tensorflow:global_step/sec: 9.89516
INFO:tensorflow:loss = 0.0585258, step = 21301 (10.106 sec)
INFO:tensorflow:global_step/sec: 10.274
INFO:tensorflow:loss = 0.182068, step = 21401 (9.733 sec)
INFO:tensorflow:global_step/sec: 10.207
INFO:tensorflow:loss = 0.15961, step = 21501 (9.797 sec)
INFO:tensorflow:global_step/sec: 10.3535
INFO:tensorflow:loss = 0.134946, step = 21601 (9.658 sec)
INFO:tensorflow:global_step/sec: 10.3885
INFO:tensorflow:loss = 0.0516901, step = 21701 (9.626 sec)
INFO:tensorflow:global_step/sec: 10.6494
INFO:tensorflow:loss = 0.0361726, step = 21801 (9.390 sec)
INFO:tensorflow:global_step/sec: 10.716
INFO:tensorflow:loss = 0.110561, step = 21901 (9.332 sec)
INFO:tensorflow:global_step/sec: 10.5541
INFO:tensorflow:loss = 0.14585, step = 22001 (9.475 sec)
INFO:tensorflow:global_step/sec: 10.4912
INFO:tensorflow:loss = 0.0292277, step = 22101 (9.532 sec)
INFO:tensorflow:global_step/sec: 10.7152
INFO:tensorflow:loss = 0.129825, step = 22201 (9.333 sec)
INFO:tensorflow:global_step/sec: 10.6897
INFO:tensorflow:loss = 0.108444, step = 22301 (9.355 sec)
INFO:tensorflow:global_step/sec: 10.2329
INFO:tensorflow:loss = 0.100388, step = 22401 (9.772 sec)
INFO:tensorflow:global_step/sec: 10.1734
INFO:tensorflow:loss = 0.0241034, step = 22501 (9.830 sec)
INFO:tensorflow:global_step/sec: 10.1079
INFO:tensorflow:loss = 0.0453775, step = 22601 (9.893 sec)
INFO:tensorflow:global_step/sec: 9.97848
INFO:tensorflow:loss = 0.251391, step = 22701 (10.022 sec)
INFO:tensorflow:global_step/sec: 10.0627
INFO:tensorflow:loss = 0.104536, step = 22801 (9.938 sec)
INFO:tensorflow:global_step/sec: 10.0783
INFO:tensorflow:loss = 0.110236, step = 22901 (9.922 sec)
INFO:tensorflow:global_step/sec: 10.0009
INFO:tensorflow:loss = 0.187455, step = 23001 (9.999 sec)
INFO:tensorflow:global_step/sec: 9.84202
INFO:tensorflow:loss = 0.146792, step = 23101 (10.160 sec)
INFO:tensorflow:global_step/sec: 9.84893
INFO:tensorflow:loss = 0.034457, step = 23201 (10.153 sec)
INFO:tensorflow:global_step/sec: 9.99413
INFO:tensorflow:loss = 0.0565486, step = 23301 (10.006 sec)
INFO:tensorflow:global_step/sec: 9.87454
INFO:tensorflow:loss = 0.0672359, step = 23401 (10.127 sec)
INFO:tensorflow:global_step/sec: 9.9731
INFO:tensorflow:loss = 0.016807, step = 23501 (10.027 sec)
INFO:tensorflow:global_step/sec: 9.70943
INFO:tensorflow:loss = 0.0304512, step = 23601 (10.299 sec)
INFO:tensorflow:global_step/sec: 9.72636
INFO:tensorflow:loss = 0.157335, step = 23701 (10.281 sec)
INFO:tensorflow:global_step/sec: 10.1877
INFO:tensorflow:loss = 0.152418, step = 23801 (9.816 sec)
INFO:tensorflow:global_step/sec: 10.4844
INFO:tensorflow:loss = 0.0916263, step = 23901 (9.538 sec)
INFO:tensorflow:global_step/sec: 10.6246
INFO:tensorflow:loss = 0.222999, step = 24001 (9.412 sec)
INFO:tensorflow:global_step/sec: 10.6499
INFO:tensorflow:loss = 0.0628916, step = 24101 (9.390 sec)
INFO:tensorflow:global_step/sec: 10.656
INFO:tensorflow:loss = 0.152039, step = 24201 (9.384 sec)
INFO:tensorflow:global_step/sec: 10.6739
INFO:tensorflow:loss = 0.364461, step = 24301 (9.369 sec)
INFO:tensorflow:global_step/sec: 10.2224
INFO:tensorflow:loss = 0.0836938, step = 24401 (9.782 sec)
INFO:tensorflow:global_step/sec: 10.3452
INFO:tensorflow:loss = 0.0618011, step = 24501 (9.666 sec)
INFO:tensorflow:global_step/sec: 10.2704
INFO:tensorflow:loss = 0.0751874, step = 24601 (9.737 sec)
INFO:tensorflow:global_step/sec: 9.75828
INFO:tensorflow:loss = 0.0735404, step = 24701 (10.248 sec)
INFO:tensorflow:global_step/sec: 9.92074
INFO:tensorflow:loss = 0.0457174, step = 24801 (10.080 sec)
INFO:tensorflow:global_step/sec: 9.41088
INFO:tensorflow:loss = 0.221096, step = 24901 (10.626 sec)
INFO:tensorflow:Saving checkpoints for 24903 into testing2/model.ckpt.
INFO:tensorflow:Starting evaluation at 2017-06-29-13:49:09
INFO:tensorflow:Restoring parameters from testing2/model.ckpt-24903
INFO:tensorflow:Evaluation [1/100]
INFO:tensorflow:Evaluation [2/100]
INFO:tensorflow:Evaluation [3/100]
INFO:tensorflow:Evaluation [4/100]
INFO:tensorflow:Evaluation [5/100]
INFO:tensorflow:Evaluation [6/100]
INFO:tensorflow:Evaluation [7/100]
INFO:tensorflow:Evaluation [8/100]
INFO:tensorflow:Evaluation [9/100]
INFO:tensorflow:Evaluation [10/100]
INFO:tensorflow:Evaluation [11/100]
INFO:tensorflow:Evaluation [12/100]
INFO:tensorflow:Evaluation [13/100]
INFO:tensorflow:Evaluation [14/100]
INFO:tensorflow:Evaluation [15/100]
INFO:tensorflow:Evaluation [16/100]
INFO:tensorflow:Evaluation [17/100]
INFO:tensorflow:Evaluation [18/100]
INFO:tensorflow:Evaluation [19/100]
INFO:tensorflow:Evaluation [20/100]
INFO:tensorflow:Evaluation [21/100]
INFO:tensorflow:Evaluation [22/100]
INFO:tensorflow:Evaluation [23/100]
INFO:tensorflow:Evaluation [24/100]
INFO:tensorflow:Evaluation [25/100]
INFO:tensorflow:Evaluation [26/100]
INFO:tensorflow:Evaluation [27/100]
INFO:tensorflow:Evaluation [28/100]
INFO:tensorflow:Evaluation [29/100]
INFO:tensorflow:Evaluation [30/100]
INFO:tensorflow:Evaluation [31/100]
INFO:tensorflow:Evaluation [32/100]
INFO:tensorflow:Evaluation [33/100]
INFO:tensorflow:Evaluation [34/100]
INFO:tensorflow:Evaluation [35/100]
INFO:tensorflow:Evaluation [36/100]
INFO:tensorflow:Evaluation [37/100]
INFO:tensorflow:Evaluation [38/100]
INFO:tensorflow:Evaluation [39/100]
INFO:tensorflow:Evaluation [40/100]
INFO:tensorflow:Evaluation [41/100]
INFO:tensorflow:Evaluation [42/100]
INFO:tensorflow:Evaluation [43/100]
INFO:tensorflow:Evaluation [44/100]
INFO:tensorflow:Evaluation [45/100]
INFO:tensorflow:Evaluation [46/100]
INFO:tensorflow:Evaluation [47/100]
INFO:tensorflow:Evaluation [48/100]
INFO:tensorflow:Evaluation [49/100]
INFO:tensorflow:Evaluation [50/100]
INFO:tensorflow:Evaluation [51/100]
INFO:tensorflow:Evaluation [52/100]
INFO:tensorflow:Evaluation [53/100]
INFO:tensorflow:Evaluation [54/100]
INFO:tensorflow:Evaluation [55/100]
INFO:tensorflow:Evaluation [56/100]
INFO:tensorflow:Evaluation [57/100]
INFO:tensorflow:Evaluation [58/100]
INFO:tensorflow:Evaluation [59/100]
INFO:tensorflow:Evaluation [60/100]
INFO:tensorflow:Evaluation [61/100]
INFO:tensorflow:Evaluation [62/100]
INFO:tensorflow:Evaluation [63/100]
INFO:tensorflow:Evaluation [64/100]
INFO:tensorflow:Evaluation [65/100]
INFO:tensorflow:Evaluation [66/100]
INFO:tensorflow:Evaluation [67/100]
INFO:tensorflow:Evaluation [68/100]
INFO:tensorflow:Evaluation [69/100]
INFO:tensorflow:Evaluation [70/100]
INFO:tensorflow:Evaluation [71/100]
INFO:tensorflow:Evaluation [72/100]
INFO:tensorflow:Evaluation [73/100]
INFO:tensorflow:Evaluation [74/100]
INFO:tensorflow:Evaluation [75/100]
INFO:tensorflow:Evaluation [76/100]
INFO:tensorflow:Evaluation [77/100]
INFO:tensorflow:Evaluation [78/100]
INFO:tensorflow:Evaluation [79/100]
INFO:tensorflow:Finished evaluation at 2017-06-29-13:49:13
INFO:tensorflow:Saving dict for global step 24903: accuracy = 0.8324, global_step = 24903, loss = 0.578522
INFO:tensorflow:Validation (step 24903): accuracy = 0.8324, global_step = 24903, loss = 0.578522
INFO:tensorflow:global_step/sec: 6.52353
INFO:tensorflow:loss = 0.0341375, step = 25001 (15.329 sec)
INFO:tensorflow:global_step/sec: 10.0441
INFO:tensorflow:loss = 0.0274761, step = 25101 (9.956 sec)
INFO:tensorflow:global_step/sec: 10.1661
INFO:tensorflow:loss = 0.300942, step = 25201 (9.837 sec)
INFO:tensorflow:global_step/sec: 10.227
INFO:tensorflow:loss = 0.0524202, step = 25301 (9.778 sec)
INFO:tensorflow:global_step/sec: 9.84267
INFO:tensorflow:loss = 0.0874858, step = 25401 (10.160 sec)
INFO:tensorflow:global_step/sec: 9.98989
INFO:tensorflow:loss = 0.380375, step = 25501 (10.010 sec)
INFO:tensorflow:global_step/sec: 10.0228
INFO:tensorflow:loss = 0.167019, step = 25601 (9.977 sec)
INFO:tensorflow:global_step/sec: 10.1635
INFO:tensorflow:loss = 0.10299, step = 25701 (9.839 sec)
INFO:tensorflow:global_step/sec: 9.97289
INFO:tensorflow:loss = 0.0480214, step = 25801 (10.027 sec)
INFO:tensorflow:global_step/sec: 10.1747
INFO:tensorflow:loss = 0.249856, step = 25901 (9.828 sec)
INFO:tensorflow:global_step/sec: 10.1208
INFO:tensorflow:loss = 0.115353, step = 26001 (9.881 sec)
INFO:tensorflow:global_step/sec: 9.98893
INFO:tensorflow:loss = 0.140886, step = 26101 (10.011 sec)
INFO:tensorflow:global_step/sec: 10.1737
INFO:tensorflow:loss = 0.0291444, step = 26201 (9.829 sec)
INFO:tensorflow:global_step/sec: 10.1633
INFO:tensorflow:loss = 0.106773, step = 26301 (9.839 sec)
INFO:tensorflow:global_step/sec: 10.1504
INFO:tensorflow:loss = 0.0279159, step = 26401 (9.852 sec)
INFO:tensorflow:global_step/sec: 10.1401
INFO:tensorflow:loss = 0.0808731, step = 26501 (9.862 sec)
INFO:tensorflow:global_step/sec: 9.90735
INFO:tensorflow:loss = 0.0298186, step = 26601 (10.094 sec)
INFO:tensorflow:global_step/sec: 10.2026
INFO:tensorflow:loss = 0.0187685, step = 26701 (9.801 sec)
INFO:tensorflow:global_step/sec: 10.243
INFO:tensorflow:loss = 0.0356779, step = 26801 (9.763 sec)
INFO:tensorflow:global_step/sec: 10.2908
INFO:tensorflow:loss = 0.0174884, step = 26901 (9.718 sec)
INFO:tensorflow:global_step/sec: 10.3543
INFO:tensorflow:loss = 0.166417, step = 27001 (9.658 sec)
INFO:tensorflow:global_step/sec: 10.6206
INFO:tensorflow:loss = 0.116242, step = 27101 (9.416 sec)
INFO:tensorflow:global_step/sec: 10.5835
INFO:tensorflow:loss = 0.0291956, step = 27201 (9.449 sec)
INFO:tensorflow:global_step/sec: 10.5545
INFO:tensorflow:loss = 0.0611862, step = 27301 (9.475 sec)
INFO:tensorflow:global_step/sec: 10.4988
INFO:tensorflow:loss = 0.0127632, step = 27401 (9.525 sec)
INFO:tensorflow:global_step/sec: 9.81971
INFO:tensorflow:loss = 0.0755489, step = 27501 (10.184 sec)
INFO:tensorflow:global_step/sec: 10.2222
INFO:tensorflow:loss = 0.0179902, step = 27601 (9.783 sec)
INFO:tensorflow:global_step/sec: 9.881
INFO:tensorflow:loss = 0.0464621, step = 27701 (10.120 sec)
INFO:tensorflow:global_step/sec: 10.2568
INFO:tensorflow:loss = 0.180232, step = 27801 (9.750 sec)
INFO:tensorflow:global_step/sec: 10.2395
INFO:tensorflow:loss = 0.0273141, step = 27901 (9.766 sec)
INFO:tensorflow:global_step/sec: 9.58837
INFO:tensorflow:loss = 0.185907, step = 28001 (10.429 sec)
INFO:tensorflow:global_step/sec: 10.0512
INFO:tensorflow:loss = 0.118873, step = 28101 (9.949 sec)
INFO:tensorflow:global_step/sec: 9.97048
INFO:tensorflow:loss = 0.0535866, step = 28201 (10.030 sec)
INFO:tensorflow:global_step/sec: 10.2256
INFO:tensorflow:loss = 0.0161786, step = 28301 (9.779 sec)
INFO:tensorflow:global_step/sec: 10.5405
INFO:tensorflow:loss = 0.0145144, step = 28401 (9.487 sec)
INFO:tensorflow:global_step/sec: 10.3069
INFO:tensorflow:loss = 0.0277693, step = 28501 (9.702 sec)










    






---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-91-c5365892903e> in <module>()
      1 # run experiment
----> 2 learn_runner.run(generate_experiment_fn(), run_config=tf.contrib.learn.RunConfig(model_dir='testing2'))

/usr/local/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py in run(experiment_fn, output_dir, schedule, run_config, hparams)
    208   schedule = schedule or _get_default_schedule(run_config)
    209 
--> 210   return _execute_schedule(experiment, schedule)
    211 
    212 

/usr/local/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py in _execute_schedule(experiment, schedule)
     45     logging.error('Allowed values for this experiment are: %s', valid_tasks)
     46     raise TypeError('Schedule references non-callable member %s' % schedule)
---> 47   return task()
     48 
     49 

/usr/local/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/experiment.py in train_and_evaluate(self)
    493             name=eval_dir_suffix, hooks=self._eval_hooks
    494         )]
--> 495       self.train(delay_secs=0)
    496 
    497     eval_result = self._call_evaluate(input_fn=self._eval_input_fn,

/usr/local/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/experiment.py in train(self, delay_secs)
    273     return self._call_train(input_fn=self._train_input_fn,
    274                             max_steps=self._train_steps,
--> 275                             hooks=self._train_monitors + extra_hooks)
    276 
    277   def evaluate(self, delay_secs=None):

/usr/local/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/experiment.py in _call_train(self, _sentinel, input_fn, steps, hooks, max_steps)
    658                                    steps=steps,
    659                                    max_steps=max_steps,
--> 660                                    hooks=hooks)
    661     else:
    662       return self._estimator.fit(input_fn=input_fn,

/usr/local/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py in train(self, input_fn, hooks, steps, max_steps)
    239       hooks.append(training.StopAtStepHook(steps, max_steps))
    240 
--> 241     loss = self._train_model(input_fn=input_fn, hooks=hooks)
    242     logging.info('Loss for final step: %s.', loss)
    243     return self

/usr/local/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py in _train_model(self, input_fn, hooks)
    610         loss = None
    611         while not mon_sess.should_stop():
--> 612           _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
    613       return loss
    614 

/usr/local/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
    503                           feed_dict=feed_dict,
    504                           options=options,
--> 505                           run_metadata=run_metadata)
    506 
    507   def should_stop(self):

/usr/local/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
    840                               feed_dict=feed_dict,
    841                               options=options,
--> 842                               run_metadata=run_metadata)
    843       except _PREEMPTION_ERRORS as e:
    844         logging.info('An error was raised. This may be due to a preemption in '

/usr/local/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, *args, **kwargs)
    796 
    797   def run(self, *args, **kwargs):
--> 798     return self._sess.run(*args, **kwargs)
    799 
    800 

/usr/local/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
    950                                   feed_dict=feed_dict,
    951                                   options=options,
--> 952                                   run_metadata=run_metadata)
    953 
    954     for hook in self._hooks:

/usr/local/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, *args, **kwargs)
    796 
    797   def run(self, *args, **kwargs):
--> 798     return self._sess.run(*args, **kwargs)
    799 
    800 

/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    787     try:
    788       result = self._run(None, fetches, feed_dict, options_ptr,
--> 789                          run_metadata_ptr)
    790       if run_metadata:
    791         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
    995     if final_fetches or final_targets:
    996       results = self._do_run(handle, final_targets, final_fetches,
--> 997                              feed_dict_string, options, run_metadata)
    998     else:
    999       results = []

/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1130     if handle is None:
   1131       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1132                            target_list, options, run_metadata)
   1133     else:
   1134       return self._do_call(_prun_fn, self._session, handle, feed_dict,

/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1137   def _do_call(self, fn, *args):
   1138     try:
-> 1139       return fn(*args)
   1140     except errors.OpError as e:
   1141       message = compat.as_text(e.message)

/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
   1119         return tf_session.TF_Run(session, options,
   1120                                  feed_dict, fetch_list, target_list,
-> 1121                                  status, run_metadata)
   1122 
   1123     def _prun_fn(session, handle, feed_dict, fetch_list):

KeyboardInterrupt:

Making Predictions

First let's generate our own sentences to see how the model classifies them.





In [188]:



    


def string_to_array(s, separator=' '):
    return s.split(separator)

def generate_data_row(sentence, label, max_length):
    sequence = np.zeros((max_length), dtype='int32')
    for i, word in enumerate(string_to_array(sentence)):
        sequence[i] = word_list.index(word)
    
    return sequence, label, max_length
    
def generate_data(sentences, labels, max_length):
    data = []
    for s, l in zip(sentences, labels):
        data.append(generate_data_row(s, l, max_length))
    
    return np.asarray(data)


sentences = ['i thought the movie was incredible and inspiring', 
             'this is a great movie',
             'this is a good movie but isnt the best',
             'it was fine i guess',
             'it was definitely bad',
             'its not that bad',
             'its not that bad i think its a good movie',
             'its not bad i think its a good movie']

labels = [[1, 0],
          [1, 0],
          [1, 0],
          [0, 1],
          [0, 1],
          [1, 0],
          [1, 0],
          [1, 0]] # [1, 0]: positive, [0, 1]: negative

my_test_data = generate_data(sentences, labels, 10)



    











In [ ]:



    






    











In [187]:



    


estimator = tf.estimator.Estimator(model_fn=model_fn,
                                   config=tf.contrib.learn.RunConfig(model_dir='tensorboard/batch_32'))

preds = estimator.predict(input_fn=get_input_fn(my_test_data, 1, 1, shuffle=False))

print()
for p, s in zip(preds, sentences):
    print('sentence:', s)
    print('good review:', p[0], 'bad review:', p[1])
    print('-' * 10)



    


















    






INFO:tensorflow:Using config: {'_save_checkpoints_steps': None, '_session_config': None, '_keep_checkpoint_every_n_hours': 10000, '_save_summary_steps': 100, '_num_ps_replicas': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f399109fd68>, '_tf_random_seed': None, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_model_dir': 'tensorboard/batch_32', '_task_id': 0, '_evaluation_master': '', '_environment': 'local', '_master': '', '_is_chief': True, '_save_checkpoints_secs': 600, '_task_type': None, '_keep_checkpoint_max': 5, '_num_worker_replicas': 0}

WARNING:tensorflow:Input graph does not contain a QueueRunner. That means predict yields forever. This is probably a mistake.
INFO:tensorflow:Restoring parameters from tensorboard/batch_32/model.ckpt-12547
sentence: i thought the movie was incredible and inspiring
good review: 0.932236 bad review: 0.0677641
----------
sentence: this is a great movie
good review: 0.980506 bad review: 0.0194938
----------
sentence: this is a good movie but isnt the best
good review: 0.926685 bad review: 0.0733154
----------
sentence: it was fine i guess
good review: 0.504088 bad review: 0.495912
----------
sentence: it was definitely bad
good review: 0.0672349 bad review: 0.932765
----------
sentence: its not that bad
good review: 0.234982 bad review: 0.765018
----------
sentence: its not that bad i think its a good movie
good review: 0.323138 bad review: 0.676862
----------
sentence: its not bad i think its a good movie
good review: 0.461568 bad review: 0.538432
----------

Content source: mari-linhares/tensorflow-workshop

Similar notebooks:

notebook.community | gallery | about