Hello, LSTM!

In this project we'd like to explore the basic usage of LSTM (Long Short-Term Memory) which is a flavor of RNN (Recurrent Neural Network).

The goals

Define the problem that LSTM can solve.
Show a basic working example of LSTM usage in Keras.
Try to learn some basic patterns in simple sequences.

Setup

Install keras, tensorflow and the basic ML/Data Science libs (numpy/matplotlib/etc.).

Set TensorFlow as the keras backend in ~/.keras/keras.json:

{"epsilon": 1e-07, "floatx": "float32", "backend": "tensorflow"}



In [177]:

    
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np

mpl.rc('image', interpolation='nearest', cmap='gray')
mpl.rc('figure', figsize=(20,10))

Basic problems

Prediction of the next value of sequence

sequence of (110)+

Just a repeated pattern:

110110110110110...

Classification of sequences

The inputs/outputs must be tensors of shape (samples, time_steps, features).

In this case (1, len(X), 1).

For simplicity we have a single training example and no test test.

Predict one step ahead:

(A, B, C, [D, E]) -> D



In [250]:

    
X = np.array([[[1],[1],[0]], [[1],[0],[1]], [[0],[1],[1]]])
y = np.array([[1], [1], [0]])

# X = np.array([[[1],[0],[0]], [[0],[1],[0]], [[0],[0],[1]]])
# y = np.array([[1], [0], [0]])



In [233]:

    
# input: 3 samples of 3-step sequences with 1 feature
# input: 3 samples with 1 feature
X.shape, y.shape









    Out[233]:





((3, 3, 1), (3, 1))

Basic usage of LTSM layers in Keras

Notes:

the first layer must specify the input shape
TensorFlow needs explicit length of series, so input_shape or batch_input_shape must be used, not just input_dim
when specifying batch_input_shape in LSTM, we need to explicitly add batch_size to model.fit()



In [4]:

    
from keras.models import Sequential
from keras.layers.core import Dense, Activation, TimeDistributedDense
from keras.layers.recurrent import LSTM









    



Using TensorFlow backend.



In [ ]:

    
# model = Sequential()
# # return_sequences=False
# model.add(LSTM(output_dim=1, input_shape=(3, 1)))
# # since the LSTM layer has only one output after activation we can directly use as model output
# model.add(Activation('sigmoid'))
# model.compile(loss='binary_crossentropy', optimizer='adam', class_mode='binary')

# This models is probably too easy and it is not able to overfit on the training dataset.
# For LSTM output dim 3 it works ok (after a few hundred epochs).



In [251]:

    
model = Sequential()
model.add(LSTM(output_dim=3, input_shape=(3, 1)))
# Since the LSTM layer has multiple outputs and model has single one
# we need to add another Dense layer with single output.
# In case the LSTM would return sequences we would use TimeDistributedDense layer.
model.add(Dense(1))
model.add(Activation('sigmoid'))



In [252]:

    
model.compile(loss='binary_crossentropy', optimizer='adam', class_mode='binary')



In [253]:

    
model.count_params()









    Out[253]:





64



In [254]:

    
model.fit(X, y, nb_epoch=500, show_accuracy=True)









    



Epoch 1/500
3/3 [==============================] - 0s - loss: 0.7282 - acc: 0.3333
Epoch 2/500
3/3 [==============================] - 0s - loss: 0.7274 - acc: 0.3333
Epoch 3/500
3/3 [==============================] - 0s - loss: 0.7265 - acc: 0.3333
Epoch 4/500
3/3 [==============================] - 0s - loss: 0.7257 - acc: 0.3333
Epoch 5/500
3/3 [==============================] - 0s - loss: 0.7249 - acc: 0.3333
Epoch 6/500
3/3 [==============================] - 0s - loss: 0.7241 - acc: 0.3333
Epoch 7/500
3/3 [==============================] - 0s - loss: 0.7233 - acc: 0.3333
Epoch 8/500
3/3 [==============================] - 0s - loss: 0.7225 - acc: 0.3333
Epoch 9/500
3/3 [==============================] - 0s - loss: 0.7217 - acc: 0.3333
Epoch 10/500
3/3 [==============================] - 0s - loss: 0.7209 - acc: 0.3333
Epoch 11/500
3/3 [==============================] - 0s - loss: 0.7201 - acc: 0.3333
Epoch 12/500
3/3 [==============================] - 0s - loss: 0.7193 - acc: 0.3333
Epoch 13/500
3/3 [==============================] - 0s - loss: 0.7186 - acc: 0.3333
Epoch 14/500
3/3 [==============================] - 0s - loss: 0.7178 - acc: 0.3333
Epoch 15/500
3/3 [==============================] - 0s - loss: 0.7170 - acc: 0.3333
Epoch 16/500
3/3 [==============================] - 0s - loss: 0.7163 - acc: 0.3333
Epoch 17/500
3/3 [==============================] - 0s - loss: 0.7155 - acc: 0.3333
Epoch 18/500
3/3 [==============================] - 0s - loss: 0.7148 - acc: 0.3333
Epoch 19/500
3/3 [==============================] - 0s - loss: 0.7140 - acc: 0.3333
Epoch 20/500
3/3 [==============================] - 0s - loss: 0.7133 - acc: 0.3333
Epoch 21/500
3/3 [==============================] - 0s - loss: 0.7125 - acc: 0.3333
Epoch 22/500
3/3 [==============================] - 0s - loss: 0.7118 - acc: 0.3333
Epoch 23/500
3/3 [==============================] - 0s - loss: 0.7111 - acc: 0.3333
Epoch 24/500
3/3 [==============================] - 0s - loss: 0.7104 - acc: 0.3333
Epoch 25/500
3/3 [==============================] - 0s - loss: 0.7096 - acc: 0.3333
Epoch 26/500
3/3 [==============================] - 0s - loss: 0.7089 - acc: 0.3333
Epoch 27/500
3/3 [==============================] - 0s - loss: 0.7082 - acc: 0.3333
Epoch 28/500
3/3 [==============================] - 0s - loss: 0.7075 - acc: 0.3333
Epoch 29/500
3/3 [==============================] - 0s - loss: 0.7068 - acc: 0.3333
Epoch 30/500
3/3 [==============================] - 0s - loss: 0.7061 - acc: 0.3333
Epoch 31/500
3/3 [==============================] - 0s - loss: 0.7054 - acc: 0.3333
Epoch 32/500
3/3 [==============================] - 0s - loss: 0.7048 - acc: 0.3333
Epoch 33/500
3/3 [==============================] - 0s - loss: 0.7041 - acc: 0.3333
Epoch 34/500
3/3 [==============================] - 0s - loss: 0.7034 - acc: 0.3333
Epoch 35/500
3/3 [==============================] - 0s - loss: 0.7027 - acc: 0.3333
Epoch 36/500
3/3 [==============================] - 0s - loss: 0.7021 - acc: 0.3333
Epoch 37/500
3/3 [==============================] - 0s - loss: 0.7014 - acc: 0.3333
Epoch 38/500
3/3 [==============================] - 0s - loss: 0.7008 - acc: 0.3333
Epoch 39/500
3/3 [==============================] - 0s - loss: 0.7001 - acc: 0.3333
Epoch 40/500
3/3 [==============================] - 0s - loss: 0.6995 - acc: 0.3333
Epoch 41/500
3/3 [==============================] - 0s - loss: 0.6988 - acc: 0.3333
Epoch 42/500
3/3 [==============================] - 0s - loss: 0.6982 - acc: 0.3333
Epoch 43/500
3/3 [==============================] - 0s - loss: 0.6976 - acc: 0.3333
Epoch 44/500
3/3 [==============================] - 0s - loss: 0.6969 - acc: 0.3333
Epoch 45/500
3/3 [==============================] - 0s - loss: 0.6963 - acc: 0.3333
Epoch 46/500
3/3 [==============================] - 0s - loss: 0.6957 - acc: 0.3333
Epoch 47/500
3/3 [==============================] - 0s - loss: 0.6951 - acc: 0.3333
Epoch 48/500
3/3 [==============================] - 0s - loss: 0.6945 - acc: 0.3333
Epoch 49/500
3/3 [==============================] - 0s - loss: 0.6938 - acc: 0.3333
Epoch 50/500
3/3 [==============================] - 0s - loss: 0.6932 - acc: 0.3333
Epoch 51/500
3/3 [==============================] - 0s - loss: 0.6926 - acc: 0.3333
Epoch 52/500
3/3 [==============================] - 0s - loss: 0.6921 - acc: 0.3333
Epoch 53/500
3/3 [==============================] - 0s - loss: 0.6915 - acc: 0.3333
Epoch 54/500
3/3 [==============================] - 0s - loss: 0.6909 - acc: 0.3333
Epoch 55/500
3/3 [==============================] - 0s - loss: 0.6903 - acc: 0.3333
Epoch 56/500
3/3 [==============================] - 0s - loss: 0.6897 - acc: 0.3333
Epoch 57/500
3/3 [==============================] - 0s - loss: 0.6891 - acc: 0.3333
Epoch 58/500
3/3 [==============================] - 0s - loss: 0.6886 - acc: 0.3333
Epoch 59/500
3/3 [==============================] - 0s - loss: 0.6880 - acc: 0.6667
Epoch 60/500
3/3 [==============================] - 0s - loss: 0.6874 - acc: 0.6667
Epoch 61/500
3/3 [==============================] - 0s - loss: 0.6869 - acc: 0.6667
Epoch 62/500
3/3 [==============================] - 0s - loss: 0.6863 - acc: 0.6667
Epoch 63/500
3/3 [==============================] - 0s - loss: 0.6858 - acc: 0.6667
Epoch 64/500
3/3 [==============================] - 0s - loss: 0.6852 - acc: 0.6667
Epoch 65/500
3/3 [==============================] - 0s - loss: 0.6847 - acc: 0.6667
Epoch 66/500
3/3 [==============================] - 0s - loss: 0.6841 - acc: 0.6667
Epoch 67/500
3/3 [==============================] - 0s - loss: 0.6836 - acc: 0.6667
Epoch 68/500
3/3 [==============================] - 0s - loss: 0.6831 - acc: 0.6667
Epoch 69/500
3/3 [==============================] - 0s - loss: 0.6825 - acc: 0.6667
Epoch 70/500
3/3 [==============================] - 0s - loss: 0.6820 - acc: 0.6667
Epoch 71/500
3/3 [==============================] - 0s - loss: 0.6815 - acc: 0.6667
Epoch 72/500
3/3 [==============================] - 0s - loss: 0.6810 - acc: 0.6667
Epoch 73/500
3/3 [==============================] - 0s - loss: 0.6804 - acc: 0.6667
Epoch 74/500
3/3 [==============================] - 0s - loss: 0.6799 - acc: 0.6667
Epoch 75/500
3/3 [==============================] - 0s - loss: 0.6794 - acc: 0.6667
Epoch 76/500
3/3 [==============================] - 0s - loss: 0.6789 - acc: 0.6667
Epoch 77/500
3/3 [==============================] - 0s - loss: 0.6784 - acc: 0.6667
Epoch 78/500
3/3 [==============================] - 0s - loss: 0.6779 - acc: 0.6667
Epoch 79/500
3/3 [==============================] - 0s - loss: 0.6774 - acc: 0.6667
Epoch 80/500
3/3 [==============================] - 0s - loss: 0.6769 - acc: 0.6667
Epoch 81/500
3/3 [==============================] - 0s - loss: 0.6764 - acc: 0.6667
Epoch 82/500
3/3 [==============================] - 0s - loss: 0.6759 - acc: 0.6667
Epoch 83/500
3/3 [==============================] - 0s - loss: 0.6754 - acc: 0.6667
Epoch 84/500
3/3 [==============================] - 0s - loss: 0.6749 - acc: 0.6667
Epoch 85/500
3/3 [==============================] - 0s - loss: 0.6744 - acc: 0.6667
Epoch 86/500
3/3 [==============================] - 0s - loss: 0.6739 - acc: 0.6667
Epoch 87/500
3/3 [==============================] - 0s - loss: 0.6735 - acc: 0.6667
Epoch 88/500
3/3 [==============================] - 0s - loss: 0.6730 - acc: 0.6667
Epoch 89/500
3/3 [==============================] - 0s - loss: 0.6725 - acc: 0.6667
Epoch 90/500
3/3 [==============================] - 0s - loss: 0.6720 - acc: 0.6667
Epoch 91/500
3/3 [==============================] - 0s - loss: 0.6716 - acc: 0.6667
Epoch 92/500
3/3 [==============================] - 0s - loss: 0.6711 - acc: 0.6667
Epoch 93/500
3/3 [==============================] - 0s - loss: 0.6706 - acc: 0.6667
Epoch 94/500
3/3 [==============================] - 0s - loss: 0.6702 - acc: 0.6667
Epoch 95/500
3/3 [==============================] - 0s - loss: 0.6697 - acc: 0.6667
Epoch 96/500
3/3 [==============================] - 0s - loss: 0.6693 - acc: 0.6667
Epoch 97/500
3/3 [==============================] - 0s - loss: 0.6688 - acc: 0.6667
Epoch 98/500
3/3 [==============================] - 0s - loss: 0.6683 - acc: 0.6667
Epoch 99/500
3/3 [==============================] - 0s - loss: 0.6679 - acc: 0.6667
Epoch 100/500
3/3 [==============================] - 0s - loss: 0.6674 - acc: 0.6667
Epoch 101/500
3/3 [==============================] - 0s - loss: 0.6670 - acc: 0.6667
Epoch 102/500
3/3 [==============================] - 0s - loss: 0.6665 - acc: 0.6667
Epoch 103/500
3/3 [==============================] - 0s - loss: 0.6661 - acc: 0.6667
Epoch 104/500
3/3 [==============================] - 0s - loss: 0.6657 - acc: 0.6667
Epoch 105/500
3/3 [==============================] - 0s - loss: 0.6652 - acc: 0.6667
Epoch 106/500
3/3 [==============================] - 0s - loss: 0.6648 - acc: 0.6667
Epoch 107/500
3/3 [==============================] - 0s - loss: 0.6644 - acc: 0.6667
Epoch 108/500
3/3 [==============================] - 0s - loss: 0.6639 - acc: 0.6667
Epoch 109/500
3/3 [==============================] - 0s - loss: 0.6635 - acc: 0.6667
Epoch 110/500
3/3 [==============================] - 0s - loss: 0.6631 - acc: 0.6667
Epoch 111/500
3/3 [==============================] - 0s - loss: 0.6626 - acc: 0.6667
Epoch 112/500
3/3 [==============================] - 0s - loss: 0.6622 - acc: 0.6667
Epoch 113/500
3/3 [==============================] - 0s - loss: 0.6618 - acc: 0.6667
Epoch 114/500
3/3 [==============================] - 0s - loss: 0.6614 - acc: 0.6667
Epoch 115/500
3/3 [==============================] - 0s - loss: 0.6609 - acc: 0.6667
Epoch 116/500
3/3 [==============================] - 0s - loss: 0.6605 - acc: 0.6667
Epoch 117/500
3/3 [==============================] - 0s - loss: 0.6601 - acc: 0.6667
Epoch 118/500
3/3 [==============================] - 0s - loss: 0.6597 - acc: 0.6667
Epoch 119/500
3/3 [==============================] - 0s - loss: 0.6593 - acc: 1.0000
Epoch 120/500
3/3 [==============================] - 0s - loss: 0.6588 - acc: 1.0000
Epoch 121/500
3/3 [==============================] - 0s - loss: 0.6584 - acc: 1.0000
Epoch 122/500
3/3 [==============================] - 0s - loss: 0.6580 - acc: 1.0000
Epoch 123/500
3/3 [==============================] - 0s - loss: 0.6576 - acc: 1.0000
Epoch 124/500
3/3 [==============================] - 0s - loss: 0.6572 - acc: 1.0000
Epoch 125/500
3/3 [==============================] - 0s - loss: 0.6568 - acc: 1.0000
Epoch 126/500
3/3 [==============================] - 0s - loss: 0.6564 - acc: 1.0000
Epoch 127/500
3/3 [==============================] - 0s - loss: 0.6560 - acc: 1.0000
Epoch 128/500
3/3 [==============================] - 0s - loss: 0.6556 - acc: 1.0000
Epoch 129/500
3/3 [==============================] - 0s - loss: 0.6552 - acc: 1.0000
Epoch 130/500
3/3 [==============================] - 0s - loss: 0.6548 - acc: 1.0000
Epoch 131/500
3/3 [==============================] - 0s - loss: 0.6544 - acc: 1.0000
Epoch 132/500
3/3 [==============================] - 0s - loss: 0.6540 - acc: 1.0000
Epoch 133/500
3/3 [==============================] - 0s - loss: 0.6536 - acc: 1.0000
Epoch 134/500
3/3 [==============================] - 0s - loss: 0.6532 - acc: 1.0000
Epoch 135/500
3/3 [==============================] - 0s - loss: 0.6528 - acc: 1.0000
Epoch 136/500
3/3 [==============================] - 0s - loss: 0.6524 - acc: 1.0000
Epoch 137/500
3/3 [==============================] - 0s - loss: 0.6520 - acc: 1.0000
Epoch 138/500
3/3 [==============================] - 0s - loss: 0.6516 - acc: 0.6667
Epoch 139/500
3/3 [==============================] - 0s - loss: 0.6512 - acc: 0.6667
Epoch 140/500
3/3 [==============================] - 0s - loss: 0.6508 - acc: 0.6667
Epoch 141/500
3/3 [==============================] - 0s - loss: 0.6505 - acc: 0.6667
Epoch 142/500
3/3 [==============================] - 0s - loss: 0.6501 - acc: 0.6667
Epoch 143/500
3/3 [==============================] - 0s - loss: 0.6497 - acc: 0.6667
Epoch 144/500
3/3 [==============================] - 0s - loss: 0.6493 - acc: 0.6667
Epoch 145/500
3/3 [==============================] - 0s - loss: 0.6489 - acc: 0.6667
Epoch 146/500
3/3 [==============================] - 0s - loss: 0.6485 - acc: 0.6667
Epoch 147/500
3/3 [==============================] - 0s - loss: 0.6481 - acc: 0.6667
Epoch 148/500
3/3 [==============================] - 0s - loss: 0.6478 - acc: 0.6667
Epoch 149/500
3/3 [==============================] - 0s - loss: 0.6474 - acc: 0.6667
Epoch 150/500
3/3 [==============================] - 0s - loss: 0.6470 - acc: 0.6667
Epoch 151/500
3/3 [==============================] - 0s - loss: 0.6466 - acc: 0.6667
Epoch 152/500
3/3 [==============================] - 0s - loss: 0.6462 - acc: 0.6667
Epoch 153/500
3/3 [==============================] - 0s - loss: 0.6459 - acc: 0.6667
Epoch 154/500
3/3 [==============================] - 0s - loss: 0.6455 - acc: 0.6667
Epoch 155/500
3/3 [==============================] - 0s - loss: 0.6451 - acc: 0.6667
Epoch 156/500
3/3 [==============================] - 0s - loss: 0.6447 - acc: 0.6667
Epoch 157/500
3/3 [==============================] - 0s - loss: 0.6443 - acc: 0.6667
Epoch 158/500
3/3 [==============================] - 0s - loss: 0.6440 - acc: 0.6667
Epoch 159/500
3/3 [==============================] - 0s - loss: 0.6436 - acc: 0.6667
Epoch 160/500
3/3 [==============================] - 0s - loss: 0.6432 - acc: 0.6667
Epoch 161/500
3/3 [==============================] - 0s - loss: 0.6428 - acc: 0.6667
Epoch 162/500
3/3 [==============================] - 0s - loss: 0.6425 - acc: 0.6667
Epoch 163/500
3/3 [==============================] - 0s - loss: 0.6421 - acc: 0.6667
Epoch 164/500
3/3 [==============================] - 0s - loss: 0.6417 - acc: 0.6667
Epoch 165/500
3/3 [==============================] - 0s - loss: 0.6413 - acc: 0.6667
Epoch 166/500
3/3 [==============================] - 0s - loss: 0.6410 - acc: 0.6667
Epoch 167/500
3/3 [==============================] - 0s - loss: 0.6406 - acc: 0.6667
Epoch 168/500
3/3 [==============================] - 0s - loss: 0.6402 - acc: 0.6667
Epoch 169/500
3/3 [==============================] - 0s - loss: 0.6398 - acc: 0.6667
Epoch 170/500
3/3 [==============================] - 0s - loss: 0.6395 - acc: 0.6667
Epoch 171/500
3/3 [==============================] - 0s - loss: 0.6391 - acc: 0.6667
Epoch 172/500
3/3 [==============================] - 0s - loss: 0.6387 - acc: 0.6667
Epoch 173/500
3/3 [==============================] - 0s - loss: 0.6383 - acc: 0.6667
Epoch 174/500
3/3 [==============================] - 0s - loss: 0.6380 - acc: 0.6667
Epoch 175/500
3/3 [==============================] - 0s - loss: 0.6376 - acc: 0.6667
Epoch 176/500
3/3 [==============================] - 0s - loss: 0.6372 - acc: 0.6667
Epoch 177/500
3/3 [==============================] - 0s - loss: 0.6369 - acc: 0.6667
Epoch 178/500
3/3 [==============================] - 0s - loss: 0.6365 - acc: 0.6667
Epoch 179/500
3/3 [==============================] - 0s - loss: 0.6361 - acc: 0.6667
Epoch 180/500
3/3 [==============================] - 0s - loss: 0.6357 - acc: 0.6667
Epoch 181/500
3/3 [==============================] - 0s - loss: 0.6354 - acc: 0.6667
Epoch 182/500
3/3 [==============================] - 0s - loss: 0.6350 - acc: 0.6667
Epoch 183/500
3/3 [==============================] - 0s - loss: 0.6346 - acc: 0.6667
Epoch 184/500
3/3 [==============================] - 0s - loss: 0.6343 - acc: 0.6667
Epoch 185/500
3/3 [==============================] - 0s - loss: 0.6339 - acc: 0.6667
Epoch 186/500
3/3 [==============================] - 0s - loss: 0.6335 - acc: 0.6667
Epoch 187/500
3/3 [==============================] - 0s - loss: 0.6331 - acc: 0.6667
Epoch 188/500
3/3 [==============================] - 0s - loss: 0.6328 - acc: 0.6667
Epoch 189/500
3/3 [==============================] - 0s - loss: 0.6324 - acc: 0.6667
Epoch 190/500
3/3 [==============================] - 0s - loss: 0.6320 - acc: 0.6667
Epoch 191/500
3/3 [==============================] - 0s - loss: 0.6316 - acc: 0.6667
Epoch 192/500
3/3 [==============================] - 0s - loss: 0.6313 - acc: 0.6667
Epoch 193/500
3/3 [==============================] - 0s - loss: 0.6309 - acc: 0.6667
Epoch 194/500
3/3 [==============================] - 0s - loss: 0.6305 - acc: 0.6667
Epoch 195/500
3/3 [==============================] - 0s - loss: 0.6301 - acc: 0.6667
Epoch 196/500
3/3 [==============================] - 0s - loss: 0.6298 - acc: 0.6667
Epoch 197/500
3/3 [==============================] - 0s - loss: 0.6294 - acc: 0.6667
Epoch 198/500
3/3 [==============================] - 0s - loss: 0.6290 - acc: 0.6667
Epoch 199/500
3/3 [==============================] - 0s - loss: 0.6286 - acc: 0.6667
Epoch 200/500
3/3 [==============================] - 0s - loss: 0.6283 - acc: 0.6667
Epoch 201/500
3/3 [==============================] - 0s - loss: 0.6279 - acc: 0.6667
Epoch 202/500
3/3 [==============================] - 0s - loss: 0.6275 - acc: 0.6667
Epoch 203/500
3/3 [==============================] - 0s - loss: 0.6271 - acc: 0.6667
Epoch 204/500
3/3 [==============================] - 0s - loss: 0.6267 - acc: 0.6667
Epoch 205/500
3/3 [==============================] - 0s - loss: 0.6264 - acc: 0.6667
Epoch 206/500
3/3 [==============================] - 0s - loss: 0.6260 - acc: 0.6667
Epoch 207/500
3/3 [==============================] - 0s - loss: 0.6256 - acc: 0.6667
Epoch 208/500
3/3 [==============================] - 0s - loss: 0.6252 - acc: 0.6667
Epoch 209/500
3/3 [==============================] - 0s - loss: 0.6248 - acc: 0.6667
Epoch 210/500
3/3 [==============================] - 0s - loss: 0.6245 - acc: 0.6667
Epoch 211/500
3/3 [==============================] - 0s - loss: 0.6241 - acc: 0.6667
Epoch 212/500
3/3 [==============================] - 0s - loss: 0.6237 - acc: 0.6667
Epoch 213/500
3/3 [==============================] - 0s - loss: 0.6233 - acc: 0.6667
Epoch 214/500
3/3 [==============================] - 0s - loss: 0.6229 - acc: 0.6667
Epoch 215/500
3/3 [==============================] - 0s - loss: 0.6225 - acc: 0.6667
Epoch 216/500
3/3 [==============================] - 0s - loss: 0.6221 - acc: 0.6667
Epoch 217/500
3/3 [==============================] - 0s - loss: 0.6217 - acc: 0.6667
Epoch 218/500
3/3 [==============================] - 0s - loss: 0.6214 - acc: 0.6667
Epoch 219/500
3/3 [==============================] - 0s - loss: 0.6210 - acc: 0.6667
Epoch 220/500
3/3 [==============================] - 0s - loss: 0.6206 - acc: 0.6667
Epoch 221/500
3/3 [==============================] - 0s - loss: 0.6202 - acc: 0.6667
Epoch 222/500
3/3 [==============================] - 0s - loss: 0.6198 - acc: 0.6667
Epoch 223/500
3/3 [==============================] - 0s - loss: 0.6194 - acc: 0.6667
Epoch 224/500
3/3 [==============================] - 0s - loss: 0.6190 - acc: 0.6667
Epoch 225/500
3/3 [==============================] - 0s - loss: 0.6186 - acc: 0.6667
Epoch 226/500
3/3 [==============================] - 0s - loss: 0.6182 - acc: 0.6667
Epoch 227/500
3/3 [==============================] - 0s - loss: 0.6178 - acc: 0.6667
Epoch 228/500
3/3 [==============================] - 0s - loss: 0.6174 - acc: 0.6667
Epoch 229/500
3/3 [==============================] - 0s - loss: 0.6170 - acc: 0.6667
Epoch 230/500
3/3 [==============================] - 0s - loss: 0.6166 - acc: 0.6667
Epoch 231/500
3/3 [==============================] - 0s - loss: 0.6162 - acc: 0.6667
Epoch 232/500
3/3 [==============================] - 0s - loss: 0.6158 - acc: 0.6667
Epoch 233/500
3/3 [==============================] - 0s - loss: 0.6154 - acc: 0.6667
Epoch 234/500
3/3 [==============================] - 0s - loss: 0.6150 - acc: 0.6667
Epoch 235/500
3/3 [==============================] - 0s - loss: 0.6146 - acc: 0.6667
Epoch 236/500
3/3 [==============================] - 0s - loss: 0.6142 - acc: 0.6667
Epoch 237/500
3/3 [==============================] - 0s - loss: 0.6138 - acc: 0.6667
Epoch 238/500
3/3 [==============================] - 0s - loss: 0.6134 - acc: 0.6667
Epoch 239/500
3/3 [==============================] - 0s - loss: 0.6130 - acc: 0.6667
Epoch 240/500
3/3 [==============================] - 0s - loss: 0.6126 - acc: 0.6667
Epoch 241/500
3/3 [==============================] - 0s - loss: 0.6122 - acc: 0.6667
Epoch 242/500
3/3 [==============================] - 0s - loss: 0.6117 - acc: 0.6667
Epoch 243/500
3/3 [==============================] - 0s - loss: 0.6113 - acc: 0.6667
Epoch 244/500
3/3 [==============================] - 0s - loss: 0.6109 - acc: 0.6667
Epoch 245/500
3/3 [==============================] - 0s - loss: 0.6105 - acc: 0.6667
Epoch 246/500
3/3 [==============================] - 0s - loss: 0.6101 - acc: 0.6667
Epoch 247/500
3/3 [==============================] - 0s - loss: 0.6097 - acc: 0.6667
Epoch 248/500
3/3 [==============================] - 0s - loss: 0.6093 - acc: 0.6667
Epoch 249/500
3/3 [==============================] - 0s - loss: 0.6088 - acc: 0.6667
Epoch 250/500
3/3 [==============================] - 0s - loss: 0.6084 - acc: 0.6667
Epoch 251/500
3/3 [==============================] - 0s - loss: 0.6080 - acc: 0.6667
Epoch 252/500
3/3 [==============================] - 0s - loss: 0.6076 - acc: 0.6667
Epoch 253/500
3/3 [==============================] - 0s - loss: 0.6071 - acc: 0.6667
Epoch 254/500
3/3 [==============================] - 0s - loss: 0.6067 - acc: 0.6667
Epoch 255/500
3/3 [==============================] - 0s - loss: 0.6063 - acc: 0.6667
Epoch 256/500
3/3 [==============================] - 0s - loss: 0.6058 - acc: 0.6667
Epoch 257/500
3/3 [==============================] - 0s - loss: 0.6054 - acc: 0.6667
Epoch 258/500
3/3 [==============================] - 0s - loss: 0.6050 - acc: 0.6667
Epoch 259/500
3/3 [==============================] - 0s - loss: 0.6045 - acc: 0.6667
Epoch 260/500
3/3 [==============================] - 0s - loss: 0.6041 - acc: 0.6667
Epoch 261/500
3/3 [==============================] - 0s - loss: 0.6037 - acc: 0.6667
Epoch 262/500
3/3 [==============================] - 0s - loss: 0.6032 - acc: 0.6667
Epoch 263/500
3/3 [==============================] - 0s - loss: 0.6028 - acc: 0.6667
Epoch 264/500
3/3 [==============================] - 0s - loss: 0.6023 - acc: 0.6667
Epoch 265/500
3/3 [==============================] - 0s - loss: 0.6019 - acc: 0.6667
Epoch 266/500
3/3 [==============================] - 0s - loss: 0.6014 - acc: 0.6667
Epoch 267/500
3/3 [==============================] - 0s - loss: 0.6010 - acc: 0.6667
Epoch 268/500
3/3 [==============================] - 0s - loss: 0.6005 - acc: 0.6667
Epoch 269/500
3/3 [==============================] - 0s - loss: 0.6001 - acc: 0.6667
Epoch 270/500
3/3 [==============================] - 0s - loss: 0.5996 - acc: 0.6667
Epoch 271/500
3/3 [==============================] - 0s - loss: 0.5992 - acc: 0.6667
Epoch 272/500
3/3 [==============================] - 0s - loss: 0.5987 - acc: 0.6667
Epoch 273/500
3/3 [==============================] - 0s - loss: 0.5983 - acc: 0.6667
Epoch 274/500
3/3 [==============================] - 0s - loss: 0.5978 - acc: 0.6667
Epoch 275/500
3/3 [==============================] - 0s - loss: 0.5973 - acc: 0.6667
Epoch 276/500
3/3 [==============================] - 0s - loss: 0.5969 - acc: 0.6667
Epoch 277/500
3/3 [==============================] - 0s - loss: 0.5964 - acc: 0.6667
Epoch 278/500
3/3 [==============================] - 0s - loss: 0.5959 - acc: 0.6667
Epoch 279/500
3/3 [==============================] - 0s - loss: 0.5955 - acc: 0.6667
Epoch 280/500
3/3 [==============================] - 0s - loss: 0.5950 - acc: 0.6667
Epoch 281/500
3/3 [==============================] - 0s - loss: 0.5945 - acc: 0.6667
Epoch 282/500
3/3 [==============================] - 0s - loss: 0.5940 - acc: 0.6667
Epoch 283/500
3/3 [==============================] - 0s - loss: 0.5935 - acc: 0.6667
Epoch 284/500
3/3 [==============================] - 0s - loss: 0.5931 - acc: 0.6667
Epoch 285/500
3/3 [==============================] - 0s - loss: 0.5926 - acc: 0.6667
Epoch 286/500
3/3 [==============================] - 0s - loss: 0.5921 - acc: 0.6667
Epoch 287/500
3/3 [==============================] - 0s - loss: 0.5916 - acc: 0.6667
Epoch 288/500
3/3 [==============================] - 0s - loss: 0.5911 - acc: 0.6667
Epoch 289/500
3/3 [==============================] - 0s - loss: 0.5906 - acc: 0.6667
Epoch 290/500
3/3 [==============================] - 0s - loss: 0.5901 - acc: 0.6667
Epoch 291/500
3/3 [==============================] - 0s - loss: 0.5896 - acc: 0.6667
Epoch 292/500
3/3 [==============================] - 0s - loss: 0.5891 - acc: 0.6667
Epoch 293/500
3/3 [==============================] - 0s - loss: 0.5886 - acc: 0.6667
Epoch 294/500
3/3 [==============================] - 0s - loss: 0.5881 - acc: 0.6667
Epoch 295/500
3/3 [==============================] - 0s - loss: 0.5876 - acc: 0.6667
Epoch 296/500
3/3 [==============================] - 0s - loss: 0.5871 - acc: 0.6667
Epoch 297/500
3/3 [==============================] - 0s - loss: 0.5866 - acc: 0.6667
Epoch 298/500
3/3 [==============================] - 0s - loss: 0.5861 - acc: 0.6667
Epoch 299/500
3/3 [==============================] - 0s - loss: 0.5855 - acc: 0.6667
Epoch 300/500
3/3 [==============================] - 0s - loss: 0.5850 - acc: 0.6667
Epoch 301/500
3/3 [==============================] - 0s - loss: 0.5845 - acc: 0.6667
Epoch 302/500
3/3 [==============================] - 0s - loss: 0.5840 - acc: 0.6667
Epoch 303/500
3/3 [==============================] - 0s - loss: 0.5834 - acc: 0.6667
Epoch 304/500
3/3 [==============================] - 0s - loss: 0.5829 - acc: 0.6667
Epoch 305/500
3/3 [==============================] - 0s - loss: 0.5824 - acc: 0.6667
Epoch 306/500
3/3 [==============================] - 0s - loss: 0.5818 - acc: 0.6667
Epoch 307/500
3/3 [==============================] - 0s - loss: 0.5813 - acc: 0.6667
Epoch 308/500
3/3 [==============================] - 0s - loss: 0.5808 - acc: 0.6667
Epoch 309/500
3/3 [==============================] - 0s - loss: 0.5802 - acc: 0.6667
Epoch 310/500
3/3 [==============================] - 0s - loss: 0.5797 - acc: 0.6667
Epoch 311/500
3/3 [==============================] - 0s - loss: 0.5791 - acc: 0.6667
Epoch 312/500
3/3 [==============================] - 0s - loss: 0.5786 - acc: 0.6667
Epoch 313/500
3/3 [==============================] - 0s - loss: 0.5780 - acc: 0.6667
Epoch 314/500
3/3 [==============================] - 0s - loss: 0.5775 - acc: 0.6667
Epoch 315/500
3/3 [==============================] - 0s - loss: 0.5769 - acc: 0.6667
Epoch 316/500
3/3 [==============================] - 0s - loss: 0.5763 - acc: 0.6667
Epoch 317/500
3/3 [==============================] - 0s - loss: 0.5758 - acc: 0.6667
Epoch 318/500
3/3 [==============================] - 0s - loss: 0.5752 - acc: 0.6667
Epoch 319/500
3/3 [==============================] - 0s - loss: 0.5746 - acc: 0.6667
Epoch 320/500
3/3 [==============================] - 0s - loss: 0.5740 - acc: 0.6667
Epoch 321/500
3/3 [==============================] - 0s - loss: 0.5735 - acc: 0.6667
Epoch 322/500
3/3 [==============================] - 0s - loss: 0.5729 - acc: 0.6667
Epoch 323/500
3/3 [==============================] - 0s - loss: 0.5723 - acc: 0.6667
Epoch 324/500
3/3 [==============================] - 0s - loss: 0.5717 - acc: 0.6667
Epoch 325/500
3/3 [==============================] - 0s - loss: 0.5711 - acc: 0.6667
Epoch 326/500
3/3 [==============================] - 0s - loss: 0.5705 - acc: 0.6667
Epoch 327/500
3/3 [==============================] - 0s - loss: 0.5699 - acc: 0.6667
Epoch 328/500
3/3 [==============================] - 0s - loss: 0.5693 - acc: 0.6667
Epoch 329/500
3/3 [==============================] - 0s - loss: 0.5687 - acc: 0.6667
Epoch 330/500
3/3 [==============================] - 0s - loss: 0.5681 - acc: 0.6667
Epoch 331/500
3/3 [==============================] - 0s - loss: 0.5675 - acc: 0.6667
Epoch 332/500
3/3 [==============================] - 0s - loss: 0.5669 - acc: 0.6667
Epoch 333/500
3/3 [==============================] - 0s - loss: 0.5663 - acc: 0.6667
Epoch 334/500
3/3 [==============================] - 0s - loss: 0.5657 - acc: 0.6667
Epoch 335/500
3/3 [==============================] - 0s - loss: 0.5651 - acc: 0.6667
Epoch 336/500
3/3 [==============================] - 0s - loss: 0.5644 - acc: 0.6667
Epoch 337/500
3/3 [==============================] - 0s - loss: 0.5638 - acc: 0.6667
Epoch 338/500
3/3 [==============================] - 0s - loss: 0.5632 - acc: 0.6667
Epoch 339/500
3/3 [==============================] - 0s - loss: 0.5626 - acc: 0.6667
Epoch 340/500
3/3 [==============================] - 0s - loss: 0.5619 - acc: 0.6667
Epoch 341/500
3/3 [==============================] - 0s - loss: 0.5613 - acc: 0.6667
Epoch 342/500
3/3 [==============================] - 0s - loss: 0.5606 - acc: 0.6667
Epoch 343/500
3/3 [==============================] - 0s - loss: 0.5600 - acc: 0.6667
Epoch 344/500
3/3 [==============================] - 0s - loss: 0.5593 - acc: 0.6667
Epoch 345/500
3/3 [==============================] - 0s - loss: 0.5587 - acc: 0.6667
Epoch 346/500
3/3 [==============================] - 0s - loss: 0.5580 - acc: 0.6667
Epoch 347/500
3/3 [==============================] - 0s - loss: 0.5574 - acc: 0.6667
Epoch 348/500
3/3 [==============================] - 0s - loss: 0.5567 - acc: 0.6667
Epoch 349/500
3/3 [==============================] - 0s - loss: 0.5560 - acc: 0.6667
Epoch 350/500
3/3 [==============================] - 0s - loss: 0.5554 - acc: 0.6667
Epoch 351/500
3/3 [==============================] - 0s - loss: 0.5547 - acc: 0.6667
Epoch 352/500
3/3 [==============================] - 0s - loss: 0.5540 - acc: 0.6667
Epoch 353/500
3/3 [==============================] - 0s - loss: 0.5533 - acc: 0.6667
Epoch 354/500
3/3 [==============================] - 0s - loss: 0.5527 - acc: 0.6667
Epoch 355/500
3/3 [==============================] - 0s - loss: 0.5520 - acc: 0.6667
Epoch 356/500
3/3 [==============================] - 0s - loss: 0.5513 - acc: 0.6667
Epoch 357/500
3/3 [==============================] - 0s - loss: 0.5506 - acc: 0.6667
Epoch 358/500
3/3 [==============================] - 0s - loss: 0.5499 - acc: 0.6667
Epoch 359/500
3/3 [==============================] - 0s - loss: 0.5492 - acc: 0.6667
Epoch 360/500
3/3 [==============================] - 0s - loss: 0.5485 - acc: 0.6667
Epoch 361/500
3/3 [==============================] - 0s - loss: 0.5478 - acc: 0.6667
Epoch 362/500
3/3 [==============================] - 0s - loss: 0.5471 - acc: 0.6667
Epoch 363/500
3/3 [==============================] - 0s - loss: 0.5464 - acc: 0.6667
Epoch 364/500
3/3 [==============================] - 0s - loss: 0.5456 - acc: 0.6667
Epoch 365/500
3/3 [==============================] - 0s - loss: 0.5449 - acc: 0.6667
Epoch 366/500
3/3 [==============================] - 0s - loss: 0.5442 - acc: 0.6667
Epoch 367/500
3/3 [==============================] - 0s - loss: 0.5434 - acc: 0.6667
Epoch 368/500
3/3 [==============================] - 0s - loss: 0.5427 - acc: 0.6667
Epoch 369/500
3/3 [==============================] - 0s - loss: 0.5420 - acc: 0.6667
Epoch 370/500
3/3 [==============================] - 0s - loss: 0.5412 - acc: 0.6667
Epoch 371/500
3/3 [==============================] - 0s - loss: 0.5405 - acc: 0.6667
Epoch 372/500
3/3 [==============================] - 0s - loss: 0.5397 - acc: 0.6667
Epoch 373/500
3/3 [==============================] - 0s - loss: 0.5390 - acc: 0.6667
Epoch 374/500
3/3 [==============================] - 0s - loss: 0.5382 - acc: 0.6667
Epoch 375/500
3/3 [==============================] - 0s - loss: 0.5375 - acc: 0.6667
Epoch 376/500
3/3 [==============================] - 0s - loss: 0.5367 - acc: 0.6667
Epoch 377/500
3/3 [==============================] - 0s - loss: 0.5359 - acc: 0.6667
Epoch 378/500
3/3 [==============================] - 0s - loss: 0.5351 - acc: 0.6667
Epoch 379/500
3/3 [==============================] - 0s - loss: 0.5344 - acc: 0.6667
Epoch 380/500
3/3 [==============================] - 0s - loss: 0.5336 - acc: 0.6667
Epoch 381/500
3/3 [==============================] - 0s - loss: 0.5328 - acc: 0.6667
Epoch 382/500
3/3 [==============================] - 0s - loss: 0.5320 - acc: 0.6667
Epoch 383/500
3/3 [==============================] - 0s - loss: 0.5312 - acc: 0.6667
Epoch 384/500
3/3 [==============================] - 0s - loss: 0.5304 - acc: 0.6667
Epoch 385/500
3/3 [==============================] - 0s - loss: 0.5296 - acc: 0.6667
Epoch 386/500
3/3 [==============================] - 0s - loss: 0.5288 - acc: 0.6667
Epoch 387/500
3/3 [==============================] - 0s - loss: 0.5280 - acc: 0.6667
Epoch 388/500
3/3 [==============================] - 0s - loss: 0.5272 - acc: 0.6667
Epoch 389/500
3/3 [==============================] - 0s - loss: 0.5264 - acc: 0.6667
Epoch 390/500
3/3 [==============================] - 0s - loss: 0.5256 - acc: 0.6667
Epoch 391/500
3/3 [==============================] - 0s - loss: 0.5248 - acc: 0.6667
Epoch 392/500
3/3 [==============================] - 0s - loss: 0.5240 - acc: 0.6667
Epoch 393/500
3/3 [==============================] - 0s - loss: 0.5232 - acc: 0.6667
Epoch 394/500
3/3 [==============================] - 0s - loss: 0.5224 - acc: 0.6667
Epoch 395/500
3/3 [==============================] - 0s - loss: 0.5215 - acc: 0.6667
Epoch 396/500
3/3 [==============================] - 0s - loss: 0.5207 - acc: 0.6667
Epoch 397/500
3/3 [==============================] - 0s - loss: 0.5199 - acc: 0.6667
Epoch 398/500
3/3 [==============================] - 0s - loss: 0.5190 - acc: 0.6667
Epoch 399/500
3/3 [==============================] - 0s - loss: 0.5182 - acc: 0.6667
Epoch 400/500
3/3 [==============================] - 0s - loss: 0.5173 - acc: 0.6667
Epoch 401/500
3/3 [==============================] - 0s - loss: 0.5165 - acc: 0.6667
Epoch 402/500
3/3 [==============================] - 0s - loss: 0.5157 - acc: 0.6667
Epoch 403/500
3/3 [==============================] - 0s - loss: 0.5150 - acc: 0.6667
Epoch 404/500
3/3 [==============================] - 0s - loss: 0.5143 - acc: 0.6667
Epoch 405/500
3/3 [==============================] - 0s - loss: 0.5136 - acc: 0.6667
Epoch 406/500
3/3 [==============================] - 0s - loss: 0.5129 - acc: 0.6667
Epoch 407/500
3/3 [==============================] - 0s - loss: 0.5121 - acc: 0.6667
Epoch 408/500
3/3 [==============================] - 0s - loss: 0.5114 - acc: 0.6667
Epoch 409/500
3/3 [==============================] - 0s - loss: 0.5106 - acc: 0.6667
Epoch 410/500
3/3 [==============================] - 0s - loss: 0.5099 - acc: 0.6667
Epoch 411/500
3/3 [==============================] - 0s - loss: 0.5091 - acc: 0.6667
Epoch 412/500
3/3 [==============================] - 0s - loss: 0.5084 - acc: 0.6667
Epoch 413/500
3/3 [==============================] - 0s - loss: 0.5076 - acc: 0.6667
Epoch 414/500
3/3 [==============================] - 0s - loss: 0.5068 - acc: 0.6667
Epoch 415/500
3/3 [==============================] - 0s - loss: 0.5061 - acc: 0.6667
Epoch 416/500
3/3 [==============================] - 0s - loss: 0.5053 - acc: 0.6667
Epoch 417/500
3/3 [==============================] - 0s - loss: 0.5045 - acc: 0.6667
Epoch 418/500
3/3 [==============================] - 0s - loss: 0.5037 - acc: 0.6667
Epoch 419/500
3/3 [==============================] - 0s - loss: 0.5029 - acc: 0.6667
Epoch 420/500
3/3 [==============================] - 0s - loss: 0.5021 - acc: 0.6667
Epoch 421/500
3/3 [==============================] - 0s - loss: 0.5013 - acc: 0.6667
Epoch 422/500
3/3 [==============================] - 0s - loss: 0.5005 - acc: 0.6667
Epoch 423/500
3/3 [==============================] - 0s - loss: 0.4997 - acc: 0.6667
Epoch 424/500
3/3 [==============================] - 0s - loss: 0.4989 - acc: 0.6667
Epoch 425/500
3/3 [==============================] - 0s - loss: 0.4981 - acc: 0.6667
Epoch 426/500
3/3 [==============================] - 0s - loss: 0.4973 - acc: 0.6667
Epoch 427/500
3/3 [==============================] - 0s - loss: 0.4965 - acc: 0.6667
Epoch 428/500
3/3 [==============================] - 0s - loss: 0.4957 - acc: 0.6667
Epoch 429/500
3/3 [==============================] - 0s - loss: 0.4949 - acc: 0.6667
Epoch 430/500
3/3 [==============================] - 0s - loss: 0.4940 - acc: 0.6667
Epoch 431/500
3/3 [==============================] - 0s - loss: 0.4932 - acc: 0.6667
Epoch 432/500
3/3 [==============================] - 0s - loss: 0.4924 - acc: 0.6667
Epoch 433/500
3/3 [==============================] - 0s - loss: 0.4915 - acc: 0.6667
Epoch 434/500
3/3 [==============================] - 0s - loss: 0.4907 - acc: 0.6667
Epoch 435/500
3/3 [==============================] - 0s - loss: 0.4899 - acc: 0.6667
Epoch 436/500
3/3 [==============================] - 0s - loss: 0.4890 - acc: 0.6667
Epoch 437/500
3/3 [==============================] - 0s - loss: 0.4882 - acc: 0.6667
Epoch 438/500
3/3 [==============================] - 0s - loss: 0.4873 - acc: 0.6667
Epoch 439/500
3/3 [==============================] - 0s - loss: 0.4865 - acc: 0.6667
Epoch 440/500
3/3 [==============================] - 0s - loss: 0.4856 - acc: 0.6667
Epoch 441/500
3/3 [==============================] - 0s - loss: 0.4848 - acc: 0.6667
Epoch 442/500
3/3 [==============================] - 0s - loss: 0.4839 - acc: 0.6667
Epoch 443/500
3/3 [==============================] - 0s - loss: 0.4830 - acc: 0.6667
Epoch 444/500
3/3 [==============================] - 0s - loss: 0.4822 - acc: 0.6667
Epoch 445/500
3/3 [==============================] - 0s - loss: 0.4813 - acc: 0.6667
Epoch 446/500
3/3 [==============================] - 0s - loss: 0.4804 - acc: 0.6667
Epoch 447/500
3/3 [==============================] - 0s - loss: 0.4795 - acc: 0.6667
Epoch 448/500
3/3 [==============================] - 0s - loss: 0.4787 - acc: 0.6667
Epoch 449/500
3/3 [==============================] - 0s - loss: 0.4778 - acc: 0.6667
Epoch 450/500
3/3 [==============================] - 0s - loss: 0.4769 - acc: 0.6667
Epoch 451/500
3/3 [==============================] - 0s - loss: 0.4760 - acc: 0.6667
Epoch 452/500
3/3 [==============================] - 0s - loss: 0.4751 - acc: 0.6667
Epoch 453/500
3/3 [==============================] - 0s - loss: 0.4742 - acc: 0.6667
Epoch 454/500
3/3 [==============================] - 0s - loss: 0.4733 - acc: 0.6667
Epoch 455/500
3/3 [==============================] - 0s - loss: 0.4723 - acc: 0.6667
Epoch 456/500
3/3 [==============================] - 0s - loss: 0.4714 - acc: 0.6667
Epoch 457/500
3/3 [==============================] - 0s - loss: 0.4705 - acc: 0.6667
Epoch 458/500
3/3 [==============================] - 0s - loss: 0.4696 - acc: 0.6667
Epoch 459/500
3/3 [==============================] - 0s - loss: 0.4686 - acc: 0.6667
Epoch 460/500
3/3 [==============================] - 0s - loss: 0.4677 - acc: 0.6667
Epoch 461/500
3/3 [==============================] - 0s - loss: 0.4667 - acc: 0.6667
Epoch 462/500
3/3 [==============================] - 0s - loss: 0.4658 - acc: 0.6667
Epoch 463/500
3/3 [==============================] - 0s - loss: 0.4648 - acc: 0.6667
Epoch 464/500
3/3 [==============================] - 0s - loss: 0.4639 - acc: 0.6667
Epoch 465/500
3/3 [==============================] - 0s - loss: 0.4629 - acc: 0.6667
Epoch 466/500
3/3 [==============================] - 0s - loss: 0.4619 - acc: 0.6667
Epoch 467/500
3/3 [==============================] - 0s - loss: 0.4610 - acc: 0.6667
Epoch 468/500
3/3 [==============================] - 0s - loss: 0.4600 - acc: 0.6667
Epoch 469/500
3/3 [==============================] - 0s - loss: 0.4590 - acc: 0.6667
Epoch 470/500
3/3 [==============================] - 0s - loss: 0.4580 - acc: 0.6667
Epoch 471/500
3/3 [==============================] - 0s - loss: 0.4570 - acc: 0.6667
Epoch 472/500
3/3 [==============================] - 0s - loss: 0.4560 - acc: 0.6667
Epoch 473/500
3/3 [==============================] - 0s - loss: 0.4551 - acc: 0.6667
Epoch 474/500
3/3 [==============================] - 0s - loss: 0.4541 - acc: 0.6667
Epoch 475/500
3/3 [==============================] - 0s - loss: 0.4531 - acc: 0.6667
Epoch 476/500
3/3 [==============================] - 0s - loss: 0.4521 - acc: 0.6667
Epoch 477/500
3/3 [==============================] - 0s - loss: 0.4511 - acc: 0.6667
Epoch 478/500
3/3 [==============================] - 0s - loss: 0.4501 - acc: 0.6667
Epoch 479/500
3/3 [==============================] - 0s - loss: 0.4491 - acc: 0.6667
Epoch 480/500
3/3 [==============================] - 0s - loss: 0.4481 - acc: 0.6667
Epoch 481/500
3/3 [==============================] - 0s - loss: 0.4471 - acc: 0.6667
Epoch 482/500
3/3 [==============================] - 0s - loss: 0.4461 - acc: 0.6667
Epoch 483/500
3/3 [==============================] - 0s - loss: 0.4451 - acc: 0.6667
Epoch 484/500
3/3 [==============================] - 0s - loss: 0.4441 - acc: 0.6667
Epoch 485/500
3/3 [==============================] - 0s - loss: 0.4430 - acc: 0.6667
Epoch 486/500
3/3 [==============================] - 0s - loss: 0.4420 - acc: 0.6667
Epoch 487/500
3/3 [==============================] - 0s - loss: 0.4409 - acc: 0.6667
Epoch 488/500
3/3 [==============================] - 0s - loss: 0.4399 - acc: 0.6667
Epoch 489/500
3/3 [==============================] - 0s - loss: 0.4388 - acc: 0.6667
Epoch 490/500
3/3 [==============================] - 0s - loss: 0.4378 - acc: 0.6667
Epoch 491/500
3/3 [==============================] - 0s - loss: 0.4367 - acc: 0.6667
Epoch 492/500
3/3 [==============================] - 0s - loss: 0.4356 - acc: 0.6667
Epoch 493/500
3/3 [==============================] - 0s - loss: 0.4346 - acc: 0.6667
Epoch 494/500
3/3 [==============================] - 0s - loss: 0.4335 - acc: 0.6667
Epoch 495/500
3/3 [==============================] - 0s - loss: 0.4324 - acc: 0.6667
Epoch 496/500
3/3 [==============================] - 0s - loss: 0.4313 - acc: 0.6667
Epoch 497/500
3/3 [==============================] - 0s - loss: 0.4302 - acc: 0.6667
Epoch 498/500
3/3 [==============================] - 0s - loss: 0.4291 - acc: 0.6667
Epoch 499/500
3/3 [==============================] - 0s - loss: 0.4280 - acc: 0.6667
Epoch 500/500
3/3 [==============================] - 0s - loss: 0.4269 - acc: 0.6667






    Out[254]:





<keras.callbacks.History at 0x12421f5c0>



In [255]:

    
plt.plot(model.predict_proba(X).flatten(), 'rx')
plt.plot(model.predict_classes(X).flatten(), 'ro')
plt.plot(y.flatten(), 'g.')
plt.xlim(-0.1, 2.1)
plt.ylim(-0.1, 1.1)









    



3/3 [==============================] - 0s
3/3 [==============================] - 0s






    Out[255]:





(-0.1, 1.1)



In [256]:

    
model.predict_proba(X)









    



3/3 [==============================] - 0s






    Out[256]:





array([[ 0.82438767],
       [ 0.69011641],
       [ 0.50994909]])



In [257]:

    
model.predict_classes(X)









    



3/3 [==============================] - 0s






    Out[257]:





array([[1],
       [1],
       [1]], dtype=int32)



In [258]:

    
# del model

LSTM weight meanings:

http://colah.github.io/posts/2015-08-Understanding-LSTMs/
source code LSTM in recurrent.py

[W_i, U_i, b_i,
 W_c, U_c, b_c,
 W_f, U_f, b_f,
 W_o, U_o, b_o]

Type of weights:

W - weight matrix - from input to output
b - bias vector - from input to output
U - weight matrix - from hidden to output (it has no companion biases)

Usage of weights:

i - input - to control whether to modify the cell state
c - candidate - a new value of cell state
f - forget - to remove the previous cell state
o - output - to control whether to output something

Inputs and outputs of a LSTM unit:

value, cell state, hidden state



In [259]:

    
weight_names = ['W_i', 'U_i', 'b_i',
 'W_c', 'U_c', 'b_c',
 'W_f', 'U_f', 'b_f',
 'W_o', 'U_o', 'b_o']



In [260]:

    
weight_shapes = [w.shape for w in model.get_weights()]
# for n, w in zip(weight_names, weight_shapes):
#     print(n, ':', w)
print(weight_shapes)









    



[(1, 3), (3, 3), (3,), (1, 3), (3, 3), (3,), (1, 3), (3, 3), (3,), (1, 3), (3, 3), (3,), (3, 1), (1,)]



In [261]:

    
def pad_vector_shape(s):
    return (s[0], 1) if len(s) == 1 else s

all_shapes = np.array([pad_vector_shape(s) for s in weight_shapes])
all_shapes









    Out[261]:





array([[1, 3],
       [3, 3],
       [3, 1],
       [1, 3],
       [3, 3],
       [3, 1],
       [1, 3],
       [3, 3],
       [3, 1],
       [1, 3],
       [3, 3],
       [3, 1],
       [3, 1],
       [1, 1]])



In [262]:

    
for w in model.get_weights():
    print(w)









    



[[-0.03112559  0.8660714   0.64888912]]
[[-0.45802712 -0.06128767 -0.89763218]
 [-0.06550674  0.03961982  1.05407453]
 [-0.4888497   0.98513061  1.33941746]]
[ 0.65779108  0.5671491   0.68662328]
[[ 0.69823277  1.03702843  1.56086171]]
[[-0.51165015 -0.2635794   0.94082028]
 [ 0.0139778  -0.04145665 -0.41632849]
 [ 0.39988595 -1.61541593  0.86261189]]
[ 0.32642099 -0.2168739  -0.63371056]
[[-0.89996403 -0.25126228  1.32094765]]
[[ 1.21803594 -1.00880325  0.51284856]
 [ 0.93082637  0.04172242  1.33407831]
 [-0.09687913 -0.89591932  0.96911085]]
[ 1.57698119  0.86581093  1.49436665]
[[-0.371564    1.31869125  1.02940512]]
[[-0.97331756  0.92339629  0.98385638]
 [-0.62391496  1.63483477  0.29820654]
 [ 0.06711823  0.35198104 -0.30188492]]
[ 0.66007471  0.26297823  0.678976  ]
[[ 1.52925777]
 [-1.38116515]
 [ 0.14560407]]
[ 0.29524684]



In [263]:

    
all_weights = np.zeros((all_shapes[:,0].sum(axis=0), all_shapes[:,1].max(axis=0)))

def add_weights(src, target):
    target[0] = src[0]
    target[1:4] = src[1]
    target[4:7,0] = src[2]
    
for i in range(4):
    add_weights(model.get_weights()[i*3:(i+1)*3], all_weights[i*7:(i+1)*7])

all_weights[28:31,0] = model.get_weights()[12].T
all_weights[31,0] = model.get_weights()[13]
    
plt.imshow(all_weights.T)


from matplotlib.patches import Rectangle

ax = plt.gca()
ax.add_patch(Rectangle([-.4, -0.4], 28-0.2, 3-0.2, fc='none', ec='r', lw=2, alpha=0.75))
ax.add_patch(Rectangle([28 - .4, -0.4], 3-0.2, 3-0.2, fc='none', ec='g', lw=2, alpha=0.75))
ax.add_patch(Rectangle([31 - .4, -0.4], 1-0.2, 3-0.2, fc='none', ec='b', lw=2, alpha=0.75))

plt.savefig('weights_110.png')



In [ ]: