Conjunto de datos

Primero, se leen las imágenes para representarlas como un conjunto lineal, en dónde cada pixel es modelado hacia un número entero.

Cabe notar que aquí hay un cambio conceptual acerca de los conjuntos, en dónde en cátedra se tiene que el conjunto de testing es el que se usa para tomar decisiones de hiperparámetros y diseño de las máquinas y es el conjunto de validación el utilizado para elegir entre múltiples máquinas o modelos.

Dado esto, y por consistencia de enunciado, se utilizará la convención dada en esta parte de la tarea


In [1]:
import pandas as pd
from sklearn.model_selection import train_test_split
def load_data():
    train = pd.read_csv('data/sign_mnist_train.csv')
    test = pd.read_csv('data/sign_mnist_test.csv')
    (train_set, validation_set) = train_test_split(train, test_size=7000, random_state=8500)
    
    y_tr = train_set['label']
    x_tr = train_set.iloc[:, 1:]
    
    y_t = test['label']
    x_t = test.iloc[:, 1:]
    
    y_v = validation_set['label']
    x_v = validation_set.iloc[:, 1:]

    return(x_tr,x_v,x_t,y_tr,y_v,y_t)
x_tr, x_v, x_t, y_tr, y_v , y_t = load_data()

Escalamiento

A continuación se presentan dos formas de escalar las imágenes


In [2]:
from sklearn import preprocessing

def scale_by_max_pixel(x_set):
    return x_set/255

def center_and_scale(x_set):
    return preprocessing.scale(x_set)

Diseño de redes neuronales

A continuación se presentan propuestas de diseño de redes neuronales para el problema y se selecciona aquel con mejor accuracy sobre el conjunto de validación

Red neuronal 1

Esta red neuronal consiste en la misma red dada en el enunciado, pero que utilizar rmsProps como optimizador


In [3]:
from keras.models import Sequential
from keras.layers import Dense, Activation
from keras.optimizers import SGD
from keras.utils.np_utils import to_categorical

# Creación de red secuencial
model1 = Sequential()
model1.add(Dense(30, input_dim=x_tr.shape[1], init='uniform', activation='relu'))
model1.add(Dense(30, init='uniform', activation='relu'))
model1.add(Dense(25, init='uniform', activation='softmax'))
model1.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
model1.fit(x_tr.values, to_categorical(y_tr), nb_epoch=100, batch_size=128, verbose=1,
validation_data=(x_v.values,to_categorical(y_v)))


Using TensorFlow backend.
/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
  return f(*args, **kwds)
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:8: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(30, input_dim=784, activation="relu", kernel_initializer="uniform")`
  
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:9: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(30, activation="relu", kernel_initializer="uniform")`
  if __name__ == '__main__':
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:10: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(25, activation="softmax", kernel_initializer="uniform")`
  # Remove the CWD from sys.path while we load stuff.
/usr/local/lib/python3.6/site-packages/keras/models.py:939: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.
  warnings.warn('The `nb_epoch` argument in `fit` '
Train on 20455 samples, validate on 7000 samples
Epoch 1/100
20455/20455 [==============================] - 1s 55us/step - loss: 3.2256 - acc: 0.0485 - val_loss: 3.1655 - val_acc: 0.0767
Epoch 2/100
20455/20455 [==============================] - 1s 43us/step - loss: 2.9841 - acc: 0.1043 - val_loss: 2.7006 - val_acc: 0.1631
Epoch 3/100
20455/20455 [==============================] - 1s 39us/step - loss: 2.6731 - acc: 0.1589 - val_loss: 2.4604 - val_acc: 0.2063
Epoch 4/100
20455/20455 [==============================] - 1s 40us/step - loss: 2.4708 - acc: 0.1963 - val_loss: 2.6592 - val_acc: 0.1767
Epoch 5/100
20455/20455 [==============================] - 1s 43us/step - loss: 2.3605 - acc: 0.2180 - val_loss: 2.1319 - val_acc: 0.2573
Epoch 6/100
20455/20455 [==============================] - 1s 38us/step - loss: 2.2687 - acc: 0.2389 - val_loss: 2.0842 - val_acc: 0.2777
Epoch 7/100
20455/20455 [==============================] - 1s 42us/step - loss: 2.2088 - acc: 0.2494 - val_loss: 2.4560 - val_acc: 0.2109
Epoch 8/100
20455/20455 [==============================] - 1s 44us/step - loss: 2.1634 - acc: 0.2555 - val_loss: 1.9677 - val_acc: 0.2957
Epoch 9/100
20455/20455 [==============================] - 1s 39us/step - loss: 2.1211 - acc: 0.2725 - val_loss: 1.9612 - val_acc: 0.2850
Epoch 10/100
20455/20455 [==============================] - 1s 40us/step - loss: 2.0919 - acc: 0.2764 - val_loss: 3.0000 - val_acc: 0.1557
Epoch 11/100
20455/20455 [==============================] - 1s 41us/step - loss: 2.0680 - acc: 0.2859 - val_loss: 2.1669 - val_acc: 0.2334
Epoch 12/100
20455/20455 [==============================] - 1s 49us/step - loss: 2.0380 - acc: 0.2947 - val_loss: 2.2235 - val_acc: 0.2690
Epoch 13/100
20455/20455 [==============================] - 1s 35us/step - loss: 2.0155 - acc: 0.2997 - val_loss: 1.9179 - val_acc: 0.3327
Epoch 14/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.9946 - acc: 0.3065 - val_loss: 1.9743 - val_acc: 0.3123
Epoch 15/100
20455/20455 [==============================] - 1s 40us/step - loss: 1.9830 - acc: 0.3132 - val_loss: 1.7849 - val_acc: 0.3599
Epoch 16/100
20455/20455 [==============================] - 1s 40us/step - loss: 1.9456 - acc: 0.3234 - val_loss: 1.7966 - val_acc: 0.3580
Epoch 17/100
20455/20455 [==============================] - 1s 41us/step - loss: 1.9441 - acc: 0.3213 - val_loss: 1.8674 - val_acc: 0.3330
Epoch 18/100
20455/20455 [==============================] - 1s 37us/step - loss: 1.9289 - acc: 0.3289 - val_loss: 2.1488 - val_acc: 0.2746
Epoch 19/100
20455/20455 [==============================] - 1s 37us/step - loss: 1.9103 - acc: 0.3332 - val_loss: 1.8559 - val_acc: 0.3400
Epoch 20/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.8938 - acc: 0.3390 - val_loss: 1.8389 - val_acc: 0.3404
Epoch 21/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.8845 - acc: 0.3352 - val_loss: 2.0081 - val_acc: 0.2980
Epoch 22/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.8714 - acc: 0.3480 - val_loss: 1.9626 - val_acc: 0.2937
Epoch 23/100
20455/20455 [==============================] - 1s 42us/step - loss: 1.8709 - acc: 0.3476 - val_loss: 1.8262 - val_acc: 0.3583
Epoch 24/100
20455/20455 [==============================] - 1s 41us/step - loss: 1.8595 - acc: 0.3511 - val_loss: 1.9482 - val_acc: 0.3226
Epoch 25/100
20455/20455 [==============================] - 1s 47us/step - loss: 1.8338 - acc: 0.3561 - val_loss: 1.7462 - val_acc: 0.3639
Epoch 26/100
20455/20455 [==============================] - 1s 40us/step - loss: 1.8348 - acc: 0.3554 - val_loss: 1.9441 - val_acc: 0.3139
Epoch 27/100
20455/20455 [==============================] - 1s 40us/step - loss: 1.8229 - acc: 0.3626 - val_loss: 1.8412 - val_acc: 0.3386
Epoch 28/100
20455/20455 [==============================] - 1s 44us/step - loss: 1.8166 - acc: 0.3554 - val_loss: 1.7028 - val_acc: 0.3837
Epoch 29/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.8022 - acc: 0.3598 - val_loss: 1.8049 - val_acc: 0.3646
Epoch 30/100
20455/20455 [==============================] - 1s 41us/step - loss: 1.7957 - acc: 0.3693 - val_loss: 1.7307 - val_acc: 0.3724
Epoch 31/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.7906 - acc: 0.3693 - val_loss: 1.6162 - val_acc: 0.4110
Epoch 32/100
20455/20455 [==============================] - 1s 40us/step - loss: 1.7765 - acc: 0.3722 - val_loss: 1.7581 - val_acc: 0.3780
Epoch 33/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.7788 - acc: 0.3719 - val_loss: 1.5904 - val_acc: 0.4164
Epoch 34/100
20455/20455 [==============================] - 1s 45us/step - loss: 1.7589 - acc: 0.3765 - val_loss: 1.6609 - val_acc: 0.4050
Epoch 35/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.7612 - acc: 0.3756 - val_loss: 1.5598 - val_acc: 0.4300
Epoch 36/100
20455/20455 [==============================] - 1s 40us/step - loss: 1.7402 - acc: 0.3784 - val_loss: 1.8586 - val_acc: 0.3424
Epoch 37/100
20455/20455 [==============================] - 1s 44us/step - loss: 1.7559 - acc: 0.3817 - val_loss: 2.6016 - val_acc: 0.2686
Epoch 38/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.7534 - acc: 0.3837 - val_loss: 2.4089 - val_acc: 0.2797
Epoch 39/100
20455/20455 [==============================] - 1s 46us/step - loss: 1.7435 - acc: 0.3838 - val_loss: 1.5603 - val_acc: 0.4346
Epoch 40/100
20455/20455 [==============================] - 1s 47us/step - loss: 1.7324 - acc: 0.3850 - val_loss: 1.5965 - val_acc: 0.4331
Epoch 41/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.7199 - acc: 0.3884 - val_loss: 1.7715 - val_acc: 0.3521
Epoch 42/100
20455/20455 [==============================] - 1s 40us/step - loss: 1.7155 - acc: 0.3922 - val_loss: 1.6561 - val_acc: 0.4126
Epoch 43/100
20455/20455 [==============================] - 1s 40us/step - loss: 1.7234 - acc: 0.3936 - val_loss: 1.6561 - val_acc: 0.4086
Epoch 44/100
20455/20455 [==============================] - 1s 40us/step - loss: 1.7059 - acc: 0.3928 - val_loss: 1.6498 - val_acc: 0.3977
Epoch 45/100
20455/20455 [==============================] - 1s 41us/step - loss: 1.7013 - acc: 0.3954 - val_loss: 1.5728 - val_acc: 0.4110
Epoch 46/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.7059 - acc: 0.3938 - val_loss: 1.5760 - val_acc: 0.4213
Epoch 47/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.6791 - acc: 0.4051 - val_loss: 1.9656 - val_acc: 0.3164
Epoch 48/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.7082 - acc: 0.3981 - val_loss: 1.5755 - val_acc: 0.4314
Epoch 49/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.6729 - acc: 0.4046 - val_loss: 1.6027 - val_acc: 0.4080
Epoch 50/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.6830 - acc: 0.3981 - val_loss: 1.5234 - val_acc: 0.4390
Epoch 51/100
20455/20455 [==============================] - 1s 37us/step - loss: 1.6791 - acc: 0.4029 - val_loss: 1.9152 - val_acc: 0.3217
Epoch 52/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.6625 - acc: 0.4113 - val_loss: 1.5886 - val_acc: 0.4407
Epoch 53/100
20455/20455 [==============================] - 1s 42us/step - loss: 1.6590 - acc: 0.4089 - val_loss: 1.5421 - val_acc: 0.4536
Epoch 54/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.6557 - acc: 0.4117 - val_loss: 1.5078 - val_acc: 0.4471
Epoch 55/100
20455/20455 [==============================] - 1s 40us/step - loss: 1.6790 - acc: 0.4046 - val_loss: 1.5657 - val_acc: 0.4223
Epoch 56/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.6483 - acc: 0.4152 - val_loss: 1.5721 - val_acc: 0.4251
Epoch 57/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.6657 - acc: 0.4115 - val_loss: 1.5703 - val_acc: 0.4417
Epoch 58/100
20455/20455 [==============================] - 1s 37us/step - loss: 1.6513 - acc: 0.4091 - val_loss: 1.7526 - val_acc: 0.3673
Epoch 59/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.6280 - acc: 0.4223 - val_loss: 1.4512 - val_acc: 0.4629
Epoch 60/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.6342 - acc: 0.4188 - val_loss: 3.0235 - val_acc: 0.2520
Epoch 61/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.6458 - acc: 0.4211 - val_loss: 2.2982 - val_acc: 0.3171
Epoch 62/100
20455/20455 [==============================] - 1s 45us/step - loss: 1.6559 - acc: 0.4141 - val_loss: 1.4950 - val_acc: 0.4529
Epoch 63/100
20455/20455 [==============================] - 1s 46us/step - loss: 1.6366 - acc: 0.4188 - val_loss: 1.4600 - val_acc: 0.4604
Epoch 64/100
20455/20455 [==============================] - 1s 42us/step - loss: 1.6414 - acc: 0.4220 - val_loss: 1.4348 - val_acc: 0.4586
Epoch 65/100
20455/20455 [==============================] - 1s 44us/step - loss: 1.6182 - acc: 0.4262 - val_loss: 1.4638 - val_acc: 0.4674
Epoch 66/100
20455/20455 [==============================] - 1s 41us/step - loss: 1.6297 - acc: 0.4216 - val_loss: 1.7552 - val_acc: 0.3746
Epoch 67/100
20455/20455 [==============================] - 1s 41us/step - loss: 1.6166 - acc: 0.4251 - val_loss: 1.5959 - val_acc: 0.4176
Epoch 68/100
20455/20455 [==============================] - 1s 41us/step - loss: 1.6217 - acc: 0.4261 - val_loss: 1.6615 - val_acc: 0.3921
Epoch 69/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.6204 - acc: 0.4242 - val_loss: 1.5456 - val_acc: 0.4341
Epoch 70/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.6110 - acc: 0.4263 - val_loss: 1.4247 - val_acc: 0.4679
Epoch 71/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.6118 - acc: 0.4258 - val_loss: 1.5021 - val_acc: 0.4479
Epoch 72/100
20455/20455 [==============================] - 1s 44us/step - loss: 1.6143 - acc: 0.4239 - val_loss: 2.0709 - val_acc: 0.3451
Epoch 73/100
20455/20455 [==============================] - 1s 41us/step - loss: 1.6128 - acc: 0.4310 - val_loss: 1.6456 - val_acc: 0.4170
Epoch 74/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.5786 - acc: 0.4365 - val_loss: 1.4905 - val_acc: 0.4431
Epoch 75/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.5901 - acc: 0.4331 - val_loss: 2.1687 - val_acc: 0.3301
Epoch 76/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.5942 - acc: 0.4353 - val_loss: 2.1217 - val_acc: 0.3479
Epoch 77/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.6104 - acc: 0.4326 - val_loss: 1.4938 - val_acc: 0.4301
Epoch 78/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.5966 - acc: 0.4332 - val_loss: 1.6852 - val_acc: 0.3816
Epoch 79/100
20455/20455 [==============================] - 1s 41us/step - loss: 1.5874 - acc: 0.4406 - val_loss: 1.4161 - val_acc: 0.4796
Epoch 80/100
20455/20455 [==============================] - 1s 43us/step - loss: 1.5789 - acc: 0.4342 - val_loss: 1.4452 - val_acc: 0.4569
Epoch 81/100
20455/20455 [==============================] - 1s 42us/step - loss: 1.5822 - acc: 0.4376 - val_loss: 1.6780 - val_acc: 0.3997
Epoch 82/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.5862 - acc: 0.4352 - val_loss: 1.6553 - val_acc: 0.4033
Epoch 83/100
20455/20455 [==============================] - 1s 40us/step - loss: 1.5775 - acc: 0.4355 - val_loss: 1.5395 - val_acc: 0.4407
Epoch 84/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.5734 - acc: 0.4391 - val_loss: 1.4179 - val_acc: 0.4616
Epoch 85/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.5668 - acc: 0.4412 - val_loss: 1.4260 - val_acc: 0.4726
Epoch 86/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.5771 - acc: 0.4408 - val_loss: 1.6817 - val_acc: 0.4056
Epoch 87/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.5895 - acc: 0.4437 - val_loss: 1.5573 - val_acc: 0.4416
Epoch 88/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.5686 - acc: 0.4377 - val_loss: 1.4196 - val_acc: 0.4717
Epoch 89/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.5698 - acc: 0.4437 - val_loss: 1.4375 - val_acc: 0.4801
Epoch 90/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.5491 - acc: 0.4426 - val_loss: 2.2615 - val_acc: 0.3380
Epoch 91/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.5619 - acc: 0.4484 - val_loss: 1.3887 - val_acc: 0.4863
Epoch 92/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.5531 - acc: 0.4501 - val_loss: 1.4191 - val_acc: 0.4790
Epoch 93/100
20455/20455 [==============================] - 1s 41us/step - loss: 1.5351 - acc: 0.4509 - val_loss: 1.3996 - val_acc: 0.4714
Epoch 94/100
20455/20455 [==============================] - 1s 42us/step - loss: 1.5478 - acc: 0.4435 - val_loss: 1.4468 - val_acc: 0.4714
Epoch 95/100
20455/20455 [==============================] - 1s 41us/step - loss: 1.5650 - acc: 0.4451 - val_loss: 1.4083 - val_acc: 0.4756
Epoch 96/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.5840 - acc: 0.4415 - val_loss: 1.4375 - val_acc: 0.4746
Epoch 97/100
20455/20455 [==============================] - 1s 41us/step - loss: 1.5527 - acc: 0.4483 - val_loss: 1.3952 - val_acc: 0.4883
Epoch 98/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.5463 - acc: 0.4515 - val_loss: 1.3455 - val_acc: 0.5013
Epoch 99/100
20455/20455 [==============================] - 1s 38us/step - loss: 1.5226 - acc: 0.4549 - val_loss: 2.0361 - val_acc: 0.3566
Epoch 100/100
20455/20455 [==============================] - 1s 39us/step - loss: 1.5367 - acc: 0.4505 - val_loss: 1.5980 - val_acc: 0.4204
Out[3]:
<keras.callbacks.History at 0x11b5cf5c0>

Con un accuracy de validación igual a 0.4204

Red neuronal 2

Esta red corresponde a una serie de capas relu con una cantidad de neuronas decreciente y qu siguen potencias de 2. Además, utiliza el mismo optimizador que la red anterior


In [4]:
from keras.models import Sequential
from keras.layers import Dense, Activation
from keras.optimizers import SGD
from keras.utils.np_utils import to_categorical

# Creación de red secuencial
model2 = Sequential()
model2.add(Dense(512, input_dim=x_tr.shape[1], init='uniform', activation='relu'))
model2.add(Dense(256, init='uniform', activation='relu'))
model2.add(Dense(128, init='uniform', activation='relu'))
model2.add(Dense(64, init='uniform', activation='relu'))
model2.add(Dense(32, init='uniform', activation='relu'))
model2.add(Dense(25, init='uniform', activation='softmax'))
model2.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
model2.fit(x_tr.values, to_categorical(y_tr), nb_epoch=100, batch_size=128, verbose=1,
validation_data=(x_v.values,to_categorical(y_v)))


/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:8: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(512, input_dim=784, activation="relu", kernel_initializer="uniform")`
  
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:9: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(256, activation="relu", kernel_initializer="uniform")`
  if __name__ == '__main__':
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:10: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(128, activation="relu", kernel_initializer="uniform")`
  # Remove the CWD from sys.path while we load stuff.
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:11: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(64, activation="relu", kernel_initializer="uniform")`
  # This is added back by InteractiveShellApp.init_path()
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:12: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(32, activation="relu", kernel_initializer="uniform")`
  if sys.path[0] == '':
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:13: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(25, activation="softmax", kernel_initializer="uniform")`
  del sys.path[0]
/usr/local/lib/python3.6/site-packages/keras/models.py:939: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.
  warnings.warn('The `nb_epoch` argument in `fit` '
Train on 20455 samples, validate on 7000 samples
Epoch 1/100
20455/20455 [==============================] - 3s 156us/step - loss: 3.0929 - acc: 0.0801 - val_loss: 3.2349 - val_acc: 0.0656
Epoch 2/100
20455/20455 [==============================] - 3s 129us/step - loss: 2.8687 - acc: 0.1212 - val_loss: 2.8490 - val_acc: 0.1046
Epoch 3/100
20455/20455 [==============================] - 3s 130us/step - loss: 2.6826 - acc: 0.1598 - val_loss: 2.3472 - val_acc: 0.2437
Epoch 4/100
20455/20455 [==============================] - 3s 127us/step - loss: 2.3854 - acc: 0.2320 - val_loss: 2.3387 - val_acc: 0.2434
Epoch 5/100
20455/20455 [==============================] - 3s 123us/step - loss: 2.1285 - acc: 0.2971 - val_loss: 1.7215 - val_acc: 0.4046
Epoch 6/100
20455/20455 [==============================] - 3s 139us/step - loss: 1.9627 - acc: 0.3415 - val_loss: 1.9858 - val_acc: 0.3321
Epoch 7/100
20455/20455 [==============================] - 3s 137us/step - loss: 1.7867 - acc: 0.3926 - val_loss: 1.5841 - val_acc: 0.4320
Epoch 8/100
20455/20455 [==============================] - 3s 132us/step - loss: 1.6465 - acc: 0.4453 - val_loss: 1.2788 - val_acc: 0.5544
Epoch 9/100
20455/20455 [==============================] - 2s 122us/step - loss: 1.4847 - acc: 0.4907 - val_loss: 2.1561 - val_acc: 0.3477
Epoch 10/100
20455/20455 [==============================] - 3s 126us/step - loss: 1.3499 - acc: 0.5402 - val_loss: 2.1166 - val_acc: 0.4193
Epoch 11/100
20455/20455 [==============================] - 3s 130us/step - loss: 1.2580 - acc: 0.5681 - val_loss: 1.0959 - val_acc: 0.5989
Epoch 12/100
20455/20455 [==============================] - 2s 117us/step - loss: 1.1368 - acc: 0.6155 - val_loss: 0.9952 - val_acc: 0.6370
Epoch 13/100
20455/20455 [==============================] - 2s 118us/step - loss: 1.0221 - acc: 0.6521 - val_loss: 0.8904 - val_acc: 0.6781
Epoch 14/100
20455/20455 [==============================] - 3s 133us/step - loss: 0.9747 - acc: 0.6804 - val_loss: 0.5603 - val_acc: 0.8207
Epoch 15/100
20455/20455 [==============================] - 3s 154us/step - loss: 0.8245 - acc: 0.7252 - val_loss: 0.6466 - val_acc: 0.7706
Epoch 16/100
20455/20455 [==============================] - 3s 148us/step - loss: 0.7992 - acc: 0.7448 - val_loss: 0.4638 - val_acc: 0.8516
Epoch 17/100
20455/20455 [==============================] - 3s 124us/step - loss: 0.7451 - acc: 0.7713 - val_loss: 0.7175 - val_acc: 0.7340
Epoch 18/100
20455/20455 [==============================] - 3s 129us/step - loss: 0.7000 - acc: 0.7906 - val_loss: 0.8524 - val_acc: 0.6944
Epoch 19/100
20455/20455 [==============================] - 3s 126us/step - loss: 0.6333 - acc: 0.8078 - val_loss: 0.2479 - val_acc: 0.9203
Epoch 20/100
20455/20455 [==============================] - 3s 131us/step - loss: 0.6095 - acc: 0.8264 - val_loss: 0.1504 - val_acc: 0.9579
Epoch 21/100
20455/20455 [==============================] - 2s 121us/step - loss: 0.5889 - acc: 0.8361 - val_loss: 2.2128 - val_acc: 0.5467
Epoch 22/100
20455/20455 [==============================] - 3s 131us/step - loss: 0.5963 - acc: 0.8387 - val_loss: 0.1584 - val_acc: 0.9549
Epoch 23/100
20455/20455 [==============================] - 3s 124us/step - loss: 0.5611 - acc: 0.8569 - val_loss: 0.3489 - val_acc: 0.8871
Epoch 24/100
20455/20455 [==============================] - 3s 134us/step - loss: 0.4660 - acc: 0.8717 - val_loss: 0.1943 - val_acc: 0.9383
Epoch 25/100
20455/20455 [==============================] - 3s 129us/step - loss: 0.5414 - acc: 0.8776 - val_loss: 0.4717 - val_acc: 0.8457
Epoch 26/100
20455/20455 [==============================] - 2s 120us/step - loss: 0.4582 - acc: 0.8796 - val_loss: 0.1883 - val_acc: 0.9347
Epoch 27/100
20455/20455 [==============================] - 3s 123us/step - loss: 0.4843 - acc: 0.8727 - val_loss: 0.2082 - val_acc: 0.9350
Epoch 28/100
20455/20455 [==============================] - 3s 132us/step - loss: 0.4527 - acc: 0.8967 - val_loss: 3.0534 - val_acc: 0.4790
Epoch 29/100
20455/20455 [==============================] - 3s 135us/step - loss: 0.5480 - acc: 0.8663 - val_loss: 0.0746 - val_acc: 0.9797
Epoch 30/100
20455/20455 [==============================] - 3s 129us/step - loss: 0.4777 - acc: 0.8875 - val_loss: 0.8733 - val_acc: 0.7916
Epoch 31/100
20455/20455 [==============================] - 3s 125us/step - loss: 0.4492 - acc: 0.8941 - val_loss: 0.0826 - val_acc: 0.9746
Epoch 32/100
20455/20455 [==============================] - 2s 122us/step - loss: 0.4371 - acc: 0.8896 - val_loss: 0.0345 - val_acc: 0.9920
Epoch 33/100
20455/20455 [==============================] - 3s 123us/step - loss: 0.4849 - acc: 0.8847 - val_loss: 0.1633 - val_acc: 0.9510
Epoch 34/100
20455/20455 [==============================] - 3s 126us/step - loss: 0.4828 - acc: 0.8869 - val_loss: 0.2075 - val_acc: 0.9319
Epoch 35/100
20455/20455 [==============================] - 3s 123us/step - loss: 0.4744 - acc: 0.8888 - val_loss: 0.2288 - val_acc: 0.9279
Epoch 36/100
20455/20455 [==============================] - 3s 124us/step - loss: 0.4224 - acc: 0.8994 - val_loss: 2.0220 - val_acc: 0.5609
Epoch 37/100
20455/20455 [==============================] - 3s 123us/step - loss: 0.5138 - acc: 0.8862 - val_loss: 0.0662 - val_acc: 0.9796
Epoch 38/100
20455/20455 [==============================] - 3s 126us/step - loss: 0.4489 - acc: 0.9018 - val_loss: 0.7502 - val_acc: 0.7813
Epoch 39/100
20455/20455 [==============================] - 3s 127us/step - loss: 0.4567 - acc: 0.8934 - val_loss: 0.9198 - val_acc: 0.7556
Epoch 40/100
20455/20455 [==============================] - 3s 128us/step - loss: 0.4353 - acc: 0.9070 - val_loss: 1.8362 - val_acc: 0.7053
Epoch 41/100
20455/20455 [==============================] - 3s 126us/step - loss: 0.4497 - acc: 0.9047 - val_loss: 0.1250 - val_acc: 0.9611
Epoch 42/100
20455/20455 [==============================] - 3s 124us/step - loss: 0.4418 - acc: 0.9060 - val_loss: 1.0557 - val_acc: 0.6677
Epoch 43/100
20455/20455 [==============================] - 3s 126us/step - loss: 0.4715 - acc: 0.8946 - val_loss: 0.6306 - val_acc: 0.8256
Epoch 44/100
20455/20455 [==============================] - 3s 127us/step - loss: 0.4057 - acc: 0.9061 - val_loss: 0.0262 - val_acc: 0.9931
Epoch 45/100
20455/20455 [==============================] - 3s 124us/step - loss: 0.5016 - acc: 0.9005 - val_loss: 0.3234 - val_acc: 0.9481
Epoch 46/100
20455/20455 [==============================] - 3s 133us/step - loss: 0.4593 - acc: 0.9086 - val_loss: 0.9424 - val_acc: 0.8474
Epoch 47/100
20455/20455 [==============================] - 3s 132us/step - loss: 0.4892 - acc: 0.9020 - val_loss: 0.9653 - val_acc: 0.7801
Epoch 48/100
20455/20455 [==============================] - 3s 132us/step - loss: 0.5354 - acc: 0.8983 - val_loss: 2.3952 - val_acc: 0.6239
Epoch 49/100
20455/20455 [==============================] - 3s 125us/step - loss: 0.5564 - acc: 0.8984 - val_loss: 0.5772 - val_acc: 0.8374
Epoch 50/100
20455/20455 [==============================] - 3s 130us/step - loss: 0.4970 - acc: 0.8945 - val_loss: 0.0783 - val_acc: 0.9781
Epoch 51/100
20455/20455 [==============================] - 3s 123us/step - loss: 0.5357 - acc: 0.9059 - val_loss: 0.0707 - val_acc: 0.9791
Epoch 52/100
20455/20455 [==============================] - 3s 123us/step - loss: 0.4923 - acc: 0.9074 - val_loss: 2.5774 - val_acc: 0.7099
Epoch 53/100
20455/20455 [==============================] - 3s 127us/step - loss: 0.5516 - acc: 0.9042 - val_loss: 0.3545 - val_acc: 0.8913
Epoch 54/100
20455/20455 [==============================] - 3s 124us/step - loss: 0.5115 - acc: 0.9109 - val_loss: 0.4073 - val_acc: 0.9100
Epoch 55/100
20455/20455 [==============================] - 3s 128us/step - loss: 0.5010 - acc: 0.9042 - val_loss: 0.0270 - val_acc: 0.9919
Epoch 56/100
20455/20455 [==============================] - 3s 129us/step - loss: 0.5936 - acc: 0.9000 - val_loss: 0.9382 - val_acc: 0.8170
Epoch 57/100
20455/20455 [==============================] - 3s 123us/step - loss: 0.5490 - acc: 0.8973 - val_loss: 0.0365 - val_acc: 0.9896
Epoch 58/100
20455/20455 [==============================] - 3s 126us/step - loss: 0.5250 - acc: 0.9007 - val_loss: 0.0908 - val_acc: 0.9681
Epoch 59/100
20455/20455 [==============================] - 2s 121us/step - loss: 0.5227 - acc: 0.9104 - val_loss: 0.2923 - val_acc: 0.9133
Epoch 60/100
20455/20455 [==============================] - 2s 121us/step - loss: 0.6411 - acc: 0.9035 - val_loss: 0.3051 - val_acc: 0.9141
Epoch 61/100
20455/20455 [==============================] - 2s 122us/step - loss: 0.5586 - acc: 0.9066 - val_loss: 0.7602 - val_acc: 0.8536
Epoch 62/100
20455/20455 [==============================] - 3s 137us/step - loss: 0.5506 - acc: 0.9072 - val_loss: 0.1438 - val_acc: 0.9714
Epoch 63/100
20455/20455 [==============================] - 3s 141us/step - loss: 0.5350 - acc: 0.9057 - val_loss: 0.7670 - val_acc: 0.8454
Epoch 64/100
20455/20455 [==============================] - 3s 137us/step - loss: 0.5321 - acc: 0.8963 - val_loss: 0.1183 - val_acc: 0.9747
Epoch 65/100
20455/20455 [==============================] - 3s 132us/step - loss: 0.7277 - acc: 0.8809 - val_loss: 0.3474 - val_acc: 0.9083
Epoch 66/100
20455/20455 [==============================] - 3s 130us/step - loss: 0.5412 - acc: 0.9013 - val_loss: 0.1553 - val_acc: 0.9599
Epoch 67/100
20455/20455 [==============================] - 3s 126us/step - loss: 0.5644 - acc: 0.9135 - val_loss: 0.0407 - val_acc: 0.9907
Epoch 68/100
20455/20455 [==============================] - 3s 140us/step - loss: 0.5944 - acc: 0.8981 - val_loss: 2.3616 - val_acc: 0.7460
Epoch 69/100
20455/20455 [==============================] - 3s 136us/step - loss: 0.5469 - acc: 0.9083 - val_loss: 0.2592 - val_acc: 0.9214
Epoch 70/100
20455/20455 [==============================] - 3s 134us/step - loss: 0.5608 - acc: 0.9100 - val_loss: 0.0825 - val_acc: 0.9777
Epoch 71/100
20455/20455 [==============================] - 3s 134us/step - loss: 0.7800 - acc: 0.8815 - val_loss: 1.1956 - val_acc: 0.7746
Epoch 72/100
20455/20455 [==============================] - 3s 132us/step - loss: 0.6086 - acc: 0.9012 - val_loss: 1.2747 - val_acc: 0.8387
Epoch 73/100
20455/20455 [==============================] - 3s 132us/step - loss: 0.6514 - acc: 0.9009 - val_loss: 0.0916 - val_acc: 0.9830
Epoch 74/100
20455/20455 [==============================] - 3s 152us/step - loss: 0.6216 - acc: 0.9071 - val_loss: 1.4906 - val_acc: 0.7983
Epoch 75/100
20455/20455 [==============================] - 3s 147us/step - loss: 0.6404 - acc: 0.8929 - val_loss: 0.1265 - val_acc: 0.9779
Epoch 76/100
20455/20455 [==============================] - 3s 145us/step - loss: 0.7844 - acc: 0.8823 - val_loss: 0.3936 - val_acc: 0.8764
Epoch 77/100
20455/20455 [==============================] - 4s 178us/step - loss: 0.6566 - acc: 0.9036 - val_loss: 0.1920 - val_acc: 0.9556
Epoch 78/100
20455/20455 [==============================] - 3s 163us/step - loss: 0.6101 - acc: 0.9081 - val_loss: 0.5951 - val_acc: 0.8923
Epoch 79/100
20455/20455 [==============================] - 3s 144us/step - loss: 0.6650 - acc: 0.8880 - val_loss: 0.0503 - val_acc: 0.9877
Epoch 80/100
20455/20455 [==============================] - 3s 139us/step - loss: 0.7719 - acc: 0.8822 - val_loss: 0.3275 - val_acc: 0.9450
Epoch 81/100
20455/20455 [==============================] - 3s 145us/step - loss: 0.5434 - acc: 0.9139 - val_loss: 0.1112 - val_acc: 0.9826
Epoch 82/100
20455/20455 [==============================] - 4s 172us/step - loss: 0.6899 - acc: 0.8995 - val_loss: 0.4497 - val_acc: 0.8947
Epoch 83/100
20455/20455 [==============================] - 3s 160us/step - loss: 0.4939 - acc: 0.9209 - val_loss: 3.4110 - val_acc: 0.5860
Epoch 84/100
20455/20455 [==============================] - 3s 153us/step - loss: 0.8814 - acc: 0.8712 - val_loss: 0.3030 - val_acc: 0.9357
Epoch 85/100
20455/20455 [==============================] - 3s 145us/step - loss: 0.8305 - acc: 0.8777 - val_loss: 0.2593 - val_acc: 0.9541
Epoch 86/100
20455/20455 [==============================] - 3s 150us/step - loss: 0.8272 - acc: 0.8795 - val_loss: 1.8142 - val_acc: 0.7980
Epoch 87/100
20455/20455 [==============================] - 3s 147us/step - loss: 0.7232 - acc: 0.8945 - val_loss: 1.7854 - val_acc: 0.7539
Epoch 88/100
20455/20455 [==============================] - 3s 142us/step - loss: 0.6973 - acc: 0.9000 - val_loss: 1.3265 - val_acc: 0.6951
Epoch 89/100
20455/20455 [==============================] - 3s 145us/step - loss: 0.7470 - acc: 0.8737 - val_loss: 0.3245 - val_acc: 0.9337
Epoch 90/100
20455/20455 [==============================] - 3s 142us/step - loss: 0.6881 - acc: 0.9020 - val_loss: 0.0784 - val_acc: 0.9874
Epoch 91/100
20455/20455 [==============================] - 3s 153us/step - loss: 0.7911 - acc: 0.8840 - val_loss: 2.4995 - val_acc: 0.7081
Epoch 92/100
20455/20455 [==============================] - 3s 140us/step - loss: 0.7233 - acc: 0.8850 - val_loss: 0.1021 - val_acc: 0.9829
Epoch 93/100
20455/20455 [==============================] - 3s 143us/step - loss: 1.1604 - acc: 0.8405 - val_loss: 0.9545 - val_acc: 0.8374
Epoch 94/100
20455/20455 [==============================] - 3s 146us/step - loss: 0.8062 - acc: 0.8585 - val_loss: 0.7526 - val_acc: 0.8827
Epoch 95/100
20455/20455 [==============================] - 3s 143us/step - loss: 1.0057 - acc: 0.8471 - val_loss: 0.8832 - val_acc: 0.8591
Epoch 96/100
20455/20455 [==============================] - 3s 146us/step - loss: 1.1447 - acc: 0.8392 - val_loss: 0.3512 - val_acc: 0.9506
Epoch 97/100
20455/20455 [==============================] - 3s 145us/step - loss: 0.9388 - acc: 0.8602 - val_loss: 0.3054 - val_acc: 0.9524
Epoch 98/100
20455/20455 [==============================] - 3s 145us/step - loss: 0.9694 - acc: 0.8621 - val_loss: 1.4981 - val_acc: 0.7693
Epoch 99/100
20455/20455 [==============================] - 3s 150us/step - loss: 0.9978 - acc: 0.8471 - val_loss: 0.6856 - val_acc: 0.8269
Epoch 100/100
20455/20455 [==============================] - 3s 145us/step - loss: 1.0873 - acc: 0.8462 - val_loss: 0.2871 - val_acc: 0.9341
Out[4]:
<keras.callbacks.History at 0x11beaeba8>

Con un accuracy de validación igual a 0.9341

Red neuronal 3

La misma red neuronal que en 2, pero con menos épocas (60).


In [5]:
from keras.models import Sequential
from keras.layers import Dense, Activation
from keras.optimizers import SGD
from keras.utils.np_utils import to_categorical

# Creación de red secuencial
model3 = Sequential()
model3.add(Dense(512, input_dim=x_tr.shape[1], init='uniform', activation='relu'))
model3.add(Dense(256, init='uniform', activation='relu'))
model3.add(Dense(128, init='uniform', activation='relu'))
model3.add(Dense(64, init='uniform', activation='relu'))
model3.add(Dense(32, init='uniform', activation='relu'))
model3.add(Dense(25, init='uniform', activation='softmax'))
model3.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
model3.fit(x_tr.values, to_categorical(y_tr), nb_epoch=60, batch_size=128, verbose=1,
validation_data=(x_v.values,to_categorical(y_v)))


/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:8: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(512, input_dim=784, activation="relu", kernel_initializer="uniform")`
  
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:9: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(256, activation="relu", kernel_initializer="uniform")`
  if __name__ == '__main__':
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:10: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(128, activation="relu", kernel_initializer="uniform")`
  # Remove the CWD from sys.path while we load stuff.
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:11: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(64, activation="relu", kernel_initializer="uniform")`
  # This is added back by InteractiveShellApp.init_path()
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:12: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(32, activation="relu", kernel_initializer="uniform")`
  if sys.path[0] == '':
/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:13: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(25, activation="softmax", kernel_initializer="uniform")`
  del sys.path[0]
/usr/local/lib/python3.6/site-packages/keras/models.py:939: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.
  warnings.warn('The `nb_epoch` argument in `fit` '
Train on 20455 samples, validate on 7000 samples
Epoch 1/60
20455/20455 [==============================] - 4s 171us/step - loss: 3.1416 - acc: 0.0644 - val_loss: 2.8436 - val_acc: 0.1370
Epoch 2/60
20455/20455 [==============================] - 3s 148us/step - loss: 2.8227 - acc: 0.1377 - val_loss: 2.6056 - val_acc: 0.1837
Epoch 3/60
20455/20455 [==============================] - 3s 155us/step - loss: 2.4672 - acc: 0.2063 - val_loss: 2.2577 - val_acc: 0.2639
Epoch 4/60
20455/20455 [==============================] - 3s 146us/step - loss: 2.2792 - acc: 0.2558 - val_loss: 2.0241 - val_acc: 0.3054
Epoch 5/60
20455/20455 [==============================] - 3s 148us/step - loss: 2.0594 - acc: 0.3173 - val_loss: 1.7244 - val_acc: 0.4110
Epoch 6/60
20455/20455 [==============================] - 3s 147us/step - loss: 1.8714 - acc: 0.3740 - val_loss: 1.7153 - val_acc: 0.4101
Epoch 7/60
20455/20455 [==============================] - 3s 143us/step - loss: 1.6893 - acc: 0.4389 - val_loss: 1.4225 - val_acc: 0.5033
Epoch 8/60
20455/20455 [==============================] - 3s 142us/step - loss: 1.4959 - acc: 0.4927 - val_loss: 1.7744 - val_acc: 0.4343
Epoch 9/60
20455/20455 [==============================] - 3s 146us/step - loss: 1.3696 - acc: 0.5389 - val_loss: 1.1492 - val_acc: 0.5943
Epoch 10/60
20455/20455 [==============================] - 3s 147us/step - loss: 1.2120 - acc: 0.5934 - val_loss: 2.0959 - val_acc: 0.4651
Epoch 11/60
20455/20455 [==============================] - 3s 148us/step - loss: 1.1335 - acc: 0.6288 - val_loss: 0.7328 - val_acc: 0.7374
Epoch 12/60
20455/20455 [==============================] - 3s 149us/step - loss: 0.9985 - acc: 0.6636 - val_loss: 0.5630 - val_acc: 0.8109
Epoch 13/60
20455/20455 [==============================] - 3s 148us/step - loss: 0.9464 - acc: 0.6994 - val_loss: 0.6230 - val_acc: 0.7659
Epoch 14/60
20455/20455 [==============================] - 3s 148us/step - loss: 0.8318 - acc: 0.7225 - val_loss: 0.3892 - val_acc: 0.8743
Epoch 15/60
20455/20455 [==============================] - 3s 151us/step - loss: 0.7639 - acc: 0.7633 - val_loss: 0.4031 - val_acc: 0.8523
Epoch 16/60
20455/20455 [==============================] - 3s 156us/step - loss: 0.7910 - acc: 0.7682 - val_loss: 0.3204 - val_acc: 0.8996
Epoch 17/60
20455/20455 [==============================] - 3s 146us/step - loss: 0.6999 - acc: 0.7959 - val_loss: 0.6494 - val_acc: 0.7761
Epoch 18/60
20455/20455 [==============================] - 3s 146us/step - loss: 0.6324 - acc: 0.8206 - val_loss: 0.3102 - val_acc: 0.8851
Epoch 19/60
20455/20455 [==============================] - 3s 141us/step - loss: 0.6035 - acc: 0.8317 - val_loss: 1.7310 - val_acc: 0.6139
Epoch 20/60
20455/20455 [==============================] - 3s 143us/step - loss: 0.6191 - acc: 0.8319 - val_loss: 0.9481 - val_acc: 0.7053
Epoch 21/60
20455/20455 [==============================] - 3s 143us/step - loss: 0.5021 - acc: 0.8604 - val_loss: 0.7262 - val_acc: 0.8044
Epoch 22/60
20455/20455 [==============================] - 3s 143us/step - loss: 0.5692 - acc: 0.8560 - val_loss: 0.3328 - val_acc: 0.8917
Epoch 23/60
20455/20455 [==============================] - 3s 149us/step - loss: 0.5279 - acc: 0.8651 - val_loss: 2.7695 - val_acc: 0.5334
Epoch 24/60
20455/20455 [==============================] - 3s 147us/step - loss: 0.4833 - acc: 0.8762 - val_loss: 0.2840 - val_acc: 0.9036
Epoch 25/60
20455/20455 [==============================] - 3s 147us/step - loss: 0.5245 - acc: 0.8781 - val_loss: 0.0803 - val_acc: 0.9757
Epoch 26/60
20455/20455 [==============================] - 3s 145us/step - loss: 0.5211 - acc: 0.8783 - val_loss: 0.8478 - val_acc: 0.7870
Epoch 27/60
20455/20455 [==============================] - 3s 141us/step - loss: 0.4583 - acc: 0.8863 - val_loss: 0.2061 - val_acc: 0.9301
Epoch 28/60
20455/20455 [==============================] - 3s 143us/step - loss: 0.4380 - acc: 0.8933 - val_loss: 2.3159 - val_acc: 0.6631
Epoch 29/60
20455/20455 [==============================] - 3s 140us/step - loss: 0.5344 - acc: 0.8830 - val_loss: 0.4824 - val_acc: 0.8431
Epoch 30/60
20455/20455 [==============================] - 3s 139us/step - loss: 0.4472 - acc: 0.8989 - val_loss: 1.2486 - val_acc: 0.6957
Epoch 31/60
20455/20455 [==============================] - 3s 138us/step - loss: 0.4652 - acc: 0.8952 - val_loss: 0.1961 - val_acc: 0.9324
Epoch 32/60
20455/20455 [==============================] - 3s 138us/step - loss: 0.4451 - acc: 0.8916 - val_loss: 0.0464 - val_acc: 0.9857
Epoch 33/60
20455/20455 [==============================] - 3s 142us/step - loss: 0.4621 - acc: 0.8997 - val_loss: 0.3624 - val_acc: 0.8987
Epoch 34/60
20455/20455 [==============================] - 3s 148us/step - loss: 0.4500 - acc: 0.8997 - val_loss: 0.0754 - val_acc: 0.9790
Epoch 35/60
20455/20455 [==============================] - 3s 144us/step - loss: 0.4330 - acc: 0.9034 - val_loss: 0.0665 - val_acc: 0.9777
Epoch 36/60
20455/20455 [==============================] - 3s 140us/step - loss: 0.4735 - acc: 0.8988 - val_loss: 0.1155 - val_acc: 0.9637
Epoch 37/60
20455/20455 [==============================] - 3s 142us/step - loss: 0.3905 - acc: 0.9155 - val_loss: 1.2710 - val_acc: 0.7691
Epoch 38/60
20455/20455 [==============================] - 3s 140us/step - loss: 0.4233 - acc: 0.9063 - val_loss: 0.0167 - val_acc: 0.9970
Epoch 39/60
20455/20455 [==============================] - 3s 139us/step - loss: 0.4381 - acc: 0.9085 - val_loss: 0.9311 - val_acc: 0.7633
Epoch 40/60
20455/20455 [==============================] - 3s 143us/step - loss: 0.4898 - acc: 0.9009 - val_loss: 0.1244 - val_acc: 0.9583
Epoch 41/60
20455/20455 [==============================] - 3s 137us/step - loss: 0.4149 - acc: 0.9087 - val_loss: 0.0170 - val_acc: 0.9979
Epoch 42/60
20455/20455 [==============================] - 3s 143us/step - loss: 0.4567 - acc: 0.9047 - val_loss: 0.4282 - val_acc: 0.8911
Epoch 43/60
20455/20455 [==============================] - 3s 147us/step - loss: 0.4170 - acc: 0.9110 - val_loss: 0.1866 - val_acc: 0.9343
Epoch 44/60
20455/20455 [==============================] - 3s 142us/step - loss: 0.4705 - acc: 0.9119 - val_loss: 3.6995 - val_acc: 0.6223
Epoch 45/60
20455/20455 [==============================] - 3s 145us/step - loss: 0.5687 - acc: 0.9004 - val_loss: 2.3344 - val_acc: 0.6227
Epoch 46/60
20455/20455 [==============================] - 3s 142us/step - loss: 0.4974 - acc: 0.9015 - val_loss: 0.3897 - val_acc: 0.8869
Epoch 47/60
20455/20455 [==============================] - 3s 141us/step - loss: 0.4379 - acc: 0.9102 - val_loss: 0.0381 - val_acc: 0.9917
Epoch 48/60
20455/20455 [==============================] - 3s 146us/step - loss: 0.4601 - acc: 0.9155 - val_loss: 2.3235 - val_acc: 0.7196
Epoch 49/60
20455/20455 [==============================] - 3s 141us/step - loss: 0.4973 - acc: 0.9120 - val_loss: 0.0349 - val_acc: 0.9907
Epoch 50/60
20455/20455 [==============================] - 3s 141us/step - loss: 0.4735 - acc: 0.9082 - val_loss: 0.1469 - val_acc: 0.9570
Epoch 51/60
20455/20455 [==============================] - 3s 140us/step - loss: 0.4889 - acc: 0.9078 - val_loss: 0.0565 - val_acc: 0.9919
Epoch 52/60
20455/20455 [==============================] - 3s 141us/step - loss: 0.5896 - acc: 0.8922 - val_loss: 0.1900 - val_acc: 0.9404
Epoch 53/60
20455/20455 [==============================] - 3s 156us/step - loss: 0.4425 - acc: 0.9231 - val_loss: 0.7172 - val_acc: 0.7884
Epoch 54/60
20455/20455 [==============================] - 3s 142us/step - loss: 0.5203 - acc: 0.9006 - val_loss: 0.4560 - val_acc: 0.8660
Epoch 55/60
20455/20455 [==============================] - 3s 142us/step - loss: 0.5149 - acc: 0.9044 - val_loss: 0.6658 - val_acc: 0.8171
Epoch 56/60
20455/20455 [==============================] - 3s 142us/step - loss: 0.4760 - acc: 0.9098 - val_loss: 0.0428 - val_acc: 0.9880
Epoch 57/60
20455/20455 [==============================] - 3s 140us/step - loss: 0.4438 - acc: 0.9163 - val_loss: 0.3101 - val_acc: 0.9083
Epoch 58/60
20455/20455 [==============================] - 3s 161us/step - loss: 0.5081 - acc: 0.9100 - val_loss: 2.7845 - val_acc: 0.6611
Epoch 59/60
20455/20455 [==============================] - 3s 150us/step - loss: 0.5151 - acc: 0.9056 - val_loss: 1.2161 - val_acc: 0.7126
Epoch 60/60
20455/20455 [==============================] - 3s 146us/step - loss: 0.5593 - acc: 0.8979 - val_loss: 1.7119 - val_acc: 0.7837
Out[5]:
<keras.callbacks.History at 0x112ec5a58>

Con un accuracy de validación igual a 0.7837

Por tanto, la mejor red neuronal corresponde a la segunda

Matriz de confusión

La matriz de confusión de la red neuronal obtenida en el tercer modelo corresponde a:


In [9]:
import itertools
import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
y_t_predict = model2.predict_classes(x_t.values)
cm = confusion_matrix(y_t, y_t_predict)
    
plt.figure(figsize=(15, 15))
plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)
plt.colorbar()
plt.show()


Se tiene que mientras más azul es el área, mayor es la dificultad de la red neuronal para clasificar correctamente las clases. Por tanto, se puede observar que a la red le cuesta más clasificar la letra 8 (h) considerando que es en dónde más se tiene conflictos al clasificar.

Cabe notar que es normal que exista una dificultad de clasificación en la diagonal debido a que es dificil distinguir una clase de si misma.

SVM no lineal

Sin procesamiento

La SVM no lineal sin procesamiento toma demasiado tiempo en converger a una solución, por lo que en lugar de buscar hiperparámetros correctos se utilizará una SVM no lineal con parámetro estándar C=0.5 y kernel tipo RBF para obtener algún resultado.

La búsqueda de parámetros se realizará en una SVM con procesamiento.


In [12]:
from sklearn.svm import SVC as SVM
from sklearn.metrics import accuracy_score
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import f1_score

def train_svm(param):
    model = SVM()
    model.set_params(C=param, kernel='rbf')
    model.fit(x_tr, y_tr)

    y_tr_pred = model.predict(x_tr)
    y_v_pred = model.predict(x_v)

    train_error = (1-accuracy_score(y_tr, y_tr_pred))
    test_error = (1-accuracy_score(y_v, y_v_pred))
    
    return (train_error, test_error)

def graph_svm_range(params):
    train_errors = []
    test_errors = []

    for depth in params:
        print('Trying param %f' % depth)
        (train, test) = train_svm(depth)
        train_errors.append(train)
        test_errors.append(test)

    plt.figure(figsize=(10, 8))
    plt.plot(params, train_errors, label="Train Error")
    plt.plot(params, test_errors, label="Test Error")
    plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
    plt.xlabel('RBF')
    plt.ylabel('Error')
    plt.show()
    
train_svm(0.5)


Out[12]:
(0.65631874847225613, 0.92414285714285715)

Con accuracy de validación de 0.08

Con procesamiento

Primero, se realizará reducción de dimensionalidad para permitir la búsqueda del mejor hiperparámetro C


In [13]:
from sklearn.decomposition import PCA
import numpy as np

x_tr_scaled = center_and_scale(x_tr)
x_t_scaled = center_and_scale(x_t)
x_v_scaled = center_and_scale(x_v)

pca_model = PCA(n_components=150)
pca_model.fit(x_tr_scaled)
x_tr_dim = pca_model.transform(x_tr_scaled)
x_t_dim = pca_model.transform(x_tr_scaled)
x_v_dim = pca_model.transform(x_v_scaled)

def train_svm(param, x_tr, y_tr, x_v, y_v):
    model = SVM()
    model.set_params(C=param, kernel='rbf')
    model.fit(x_tr, y_tr)

    y_tr_pred = model.predict(x_tr)
    y_v_pred = model.predict(x_v)

    train_error = (1-accuracy_score(y_tr, y_tr_pred))
    test_error = (1-accuracy_score(y_v, y_v_pred))
    
    return (train_error, test_error)

def graph_svm_range(params, x_tr, y_tr, x_v, y_v):
    train_errors = []
    test_errors = []

    for depth in params:
        print('Trying param %f' % depth)
        (train, test) = train_svm(depth, x_tr, y_tr, x_v, y_v)
        train_errors.append(train)
        test_errors.append(test)

    plt.figure(figsize=(10, 8))
    plt.plot(params, train_errors, label="Train Error")
    plt.plot(params, test_errors, label="Test Error")
    plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
    plt.xlabel('RBF')
    plt.ylabel('Error')
    plt.show()

params = np.arange(0.001, 1.0, 0.25)
graph_svm_range(params, x_tr_dim, y_tr, x_v_dim, y_v)


Trying param 0.001000
Trying param 0.251000
Trying param 0.501000
Trying param 0.751000

In [20]:
train_svm(0.75, x_tr_dim, y_tr, x_v_dim, y_v)


Out[20]:
(0.0, 0.001571428571428557)

Que se traduce a un accuracy de validación de 0.9991. Esto indica un posible overfit a los datos de validación, por lo que es de esperar un mal rendimiento al momento de probar los datos de testing.

Árbol de clasificación

Sin procesamiento

In [7]:
from sklearn.tree import DecisionTreeClassifier as Tree
from sklearn.metrics import accuracy_score
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import f1_score

def train_tree(depth, x_tr, y_tr, x_v, y_v):
    # Entrenar el árbol
    model = Tree()
    model.set_params(max_depth=depth, criterion='gini', splitter='best')
    model.fit(x_tr, y_tr)

    y_tr_pred = model.predict(x_tr)
    y_v_pred = model.predict(x_v)


    train_error = (accuracy_score(y_tr, y_tr_pred))
    test_error = (accuracy_score(y_v, y_v_pred))

    return (train_error, test_error)

def graph_tree_range(params, x_tr, y_tr, x_v, y_v):
    train_errors = []
    test_errors = []

    for depth in params:
        print('Trying depth %d' % depth)
        (train, test) = train_tree(depth, x_tr, y_tr, x_v, y_v)
        train_errors.append(train)
        test_errors.append(test)

    plt.figure(figsize=(10, 8))
    plt.plot(params, train_errors, label="Train accuracy")
    plt.plot(params, test_errors, label="Validation accuracy")
    plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
    plt.xlabel('Profundidad del árbol')
    plt.ylabel('Error')
    plt.show()
    
params = np.arange(1, 30, 1)
graph_tree_range(params, x_tr, y_tr, x_v, y_v)


Trying depth 1
Trying depth 2
Trying depth 3
Trying depth 4
Trying depth 5
Trying depth 6
Trying depth 7
Trying depth 8
Trying depth 9
Trying depth 10
Trying depth 11
Trying depth 12
Trying depth 13
Trying depth 14
Trying depth 15
Trying depth 16
Trying depth 17
Trying depth 18
Trying depth 19
Trying depth 20
Trying depth 21
Trying depth 22
Trying depth 23
Trying depth 24
Trying depth 25
Trying depth 26
Trying depth 27
Trying depth 28
Trying depth 29
Con procesamiento

In [8]:
x_tr_scaled = center_and_scale(x_tr)
x_t_scaled = center_and_scale(x_t)
x_v_scaled = center_and_scale(x_v)

params = np.arange(1, 30, 1)
graph_tree_range(params, x_tr_scaled, y_tr, x_v_scaled, y_v)


Trying depth 1
Trying depth 2
Trying depth 3
Trying depth 4
Trying depth 5
Trying depth 6
Trying depth 7
Trying depth 8
Trying depth 9
Trying depth 10
Trying depth 11
Trying depth 12
Trying depth 13
Trying depth 14
Trying depth 15
Trying depth 16
Trying depth 17
Trying depth 18
Trying depth 19
Trying depth 20
Trying depth 21
Trying depth 22
Trying depth 23
Trying depth 24
Trying depth 25
Trying depth 26
Trying depth 27
Trying depth 28
Trying depth 29

Selección de mejor modelo

Utilizando cada máquina con los mejores hiperparámetros y considerando datos con pre procesamiento, se tiene que el mejor modelo estará dado por el accuracy de los datos de testing. Se tiene:


In [26]:
from sklearn.metrics import accuracy_score

# Neural network
y_t_pred = model3.predict_classes(x_t.values)
print("Accuracy red neuronal: %f" % accuracy_score(y_t, y_t_pred))

tree_accuracy = train_tree(30, x_tr_scaled, y_tr, x_t_scaled, y_t)
print("Accuracy decision tree: %f" % tree_accuracy[1])

x_t_dim = pca_model.transform(x_t_scaled)
svm_errors = train_svm(0.75, x_tr_dim, y_tr, x_t_dim, y_t)
svm_errors


Accuracy red neuronal: 0.606247
Accuracy decision tree: 0.424289
Out[26]:
(0.0, 0.19283324037925265)

In [31]:
svm_acc = (svm_errors[1] - 1)*-1
print("Accuracy SVM: %f" % svm_acc)


Accuracy SVM: 0.807167

La SVM lineal con kernel RBF presenta el mejor error de validación para este caso. Sin embargo, esto es sobre el conjunto de testing transformado a la representación con dimensionalidad reducida. Por esto, es muy probable que este modelo no sirva para generalizar las imágenes y es muy probable también que estemos en un caso de overfitting.

Por esto, es mejor seleccionar la redes neuronales o diseñar una nueva estrategia para mejorar la SVM. Esto es dificil considerando los problemas de rendimiento que presenta SVM al tener muchas opciones de clasificación ya que debe ajustar los hiperplanos para satisfacer demasiados constraints.

Por temas de capacidad computacional, es entonces aún más recomendable utilizar redes neuronales, ya que pueden llegar a resultados más verídicos en menos tiempo.