notebook.community

Edit and run

1. Giới thiệu

Trong notebook này, mình sẽ trình bày cách giải quyết đề tài tuyển dụng của VinID. Mô hình CNN được sử dụng để phân loại 10 số viết tay trong bộ MNIST. Trong notebook này,bao gồm các phần sau:

  • 1. Giới thiệu
  • 2. Tiền xử lý dữ liệu
    • 2.1 Load dữ liệu
    • 2.2 Kiểm tra missing value
    • 2.3 Chuẩn hóa
    • 2.5 Label encoding
    • 2.6 Xây dựng tập train/test
  • 3. Data augmentation
  • 4. Xây dựng mô hình
    • 4.1 Xây dựng mô hình
  • 5. Tunning parameter
    • 5.1 Khai báo không gian tìm kiếm siêu tham số
    • 5.2 Grid search
  • 6. So sánh các optimizer và loss function
    • 6.1 So sánh optimizer
    • 6.2 So sánh loss function
  • 7. Đánh giá mô hình
    • 7.1 Confusion matrix
  • 8. Dự đoán
    • 8.1 Predict and Submit results

Cài đặt thư viện hyperas để hỗ trỡ quá trình tunning siêu tham số. Hyperas cung cấp các api rất tiện lợi cho quá trình theo huấn luyện và theo dõi độ chính xác của model tại mỗi bộ tham số.





In [1]:



    


!pip install hyperas



    


















    






Requirement already satisfied: hyperas in /root/anaconda3/lib/python3.6/site-packages (0.4.1)
Requirement already satisfied: entrypoints in /root/anaconda3/lib/python3.6/site-packages (from hyperas) (0.2.3)
Requirement already satisfied: keras in /root/anaconda3/lib/python3.6/site-packages (from hyperas) (2.2.4)
Requirement already satisfied: hyperopt in /root/anaconda3/lib/python3.6/site-packages (from hyperas) (0.1.2)
Requirement already satisfied: nbformat in /root/anaconda3/lib/python3.6/site-packages (from hyperas) (4.4.0)
Requirement already satisfied: jupyter in /root/anaconda3/lib/python3.6/site-packages (from hyperas) (1.0.0)
Requirement already satisfied: nbconvert in /root/anaconda3/lib/python3.6/site-packages (from hyperas) (5.3.1)
Requirement already satisfied: scipy>=0.14 in /root/anaconda3/lib/python3.6/site-packages (from keras->hyperas) (1.3.0)
Requirement already satisfied: keras-applications>=1.0.6 in /root/.local/lib/python3.6/site-packages (from keras->hyperas) (1.0.6)
Requirement already satisfied: keras-preprocessing>=1.0.5 in /root/.local/lib/python3.6/site-packages (from keras->hyperas) (1.0.5)
Requirement already satisfied: pyyaml in /root/anaconda3/lib/python3.6/site-packages (from keras->hyperas) (3.12)
Requirement already satisfied: numpy>=1.9.1 in /root/.local/lib/python3.6/site-packages (from keras->hyperas) (1.17.0)
Requirement already satisfied: six>=1.9.0 in /root/.local/lib/python3.6/site-packages (from keras->hyperas) (1.12.0)
Requirement already satisfied: h5py in /root/anaconda3/lib/python3.6/site-packages (from keras->hyperas) (2.7.1)
Requirement already satisfied: tqdm in /root/anaconda3/lib/python3.6/site-packages (from hyperopt->hyperas) (4.32.2)
Requirement already satisfied: pymongo in /root/anaconda3/lib/python3.6/site-packages (from hyperopt->hyperas) (3.9.0)
Requirement already satisfied: future in /root/.local/lib/python3.6/site-packages (from hyperopt->hyperas) (0.17.1)
Requirement already satisfied: networkx in /root/anaconda3/lib/python3.6/site-packages (from hyperopt->hyperas) (2.1)
Requirement already satisfied: ipython_genutils in /root/anaconda3/lib/python3.6/site-packages (from nbformat->hyperas) (0.2.0)
Requirement already satisfied: traitlets>=4.1 in /root/anaconda3/lib/python3.6/site-packages (from nbformat->hyperas) (4.3.2)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /root/anaconda3/lib/python3.6/site-packages (from nbformat->hyperas) (2.6.0)
Requirement already satisfied: jupyter_core in /root/anaconda3/lib/python3.6/site-packages (from nbformat->hyperas) (4.4.0)
Requirement already satisfied: notebook in /root/anaconda3/lib/python3.6/site-packages (from jupyter->hyperas) (5.5.0)
Requirement already satisfied: qtconsole in /root/anaconda3/lib/python3.6/site-packages (from jupyter->hyperas) (4.3.1)
Requirement already satisfied: jupyter-console in /root/anaconda3/lib/python3.6/site-packages (from jupyter->hyperas) (5.2.0)
Requirement already satisfied: ipykernel in /root/anaconda3/lib/python3.6/site-packages (from jupyter->hyperas) (4.8.2)
Requirement already satisfied: ipywidgets in /root/anaconda3/lib/python3.6/site-packages (from jupyter->hyperas) (7.2.1)
Requirement already satisfied: mistune>=0.7.4 in /root/anaconda3/lib/python3.6/site-packages (from nbconvert->hyperas) (0.8.3)
Requirement already satisfied: jinja2 in /root/anaconda3/lib/python3.6/site-packages (from nbconvert->hyperas) (2.10)
Requirement already satisfied: pygments in /root/anaconda3/lib/python3.6/site-packages (from nbconvert->hyperas) (2.2.0)
Requirement already satisfied: bleach in /root/anaconda3/lib/python3.6/site-packages (from nbconvert->hyperas) (2.1.3)
Requirement already satisfied: pandocfilters>=1.4.1 in /root/anaconda3/lib/python3.6/site-packages (from nbconvert->hyperas) (1.4.2)
Requirement already satisfied: testpath in /root/anaconda3/lib/python3.6/site-packages (from nbconvert->hyperas) (0.3.1)
Requirement already satisfied: decorator>=4.1.0 in /root/anaconda3/lib/python3.6/site-packages (from networkx->hyperopt->hyperas) (4.3.0)
Requirement already satisfied: Send2Trash in /root/anaconda3/lib/python3.6/site-packages (from notebook->jupyter->hyperas) (1.5.0)
Requirement already satisfied: pyzmq>=17 in /root/anaconda3/lib/python3.6/site-packages (from notebook->jupyter->hyperas) (17.0.0)
Requirement already satisfied: terminado>=0.8.1 in /root/anaconda3/lib/python3.6/site-packages (from notebook->jupyter->hyperas) (0.8.1)
Requirement already satisfied: jupyter-client>=5.2.0 in /root/anaconda3/lib/python3.6/site-packages (from notebook->jupyter->hyperas) (5.2.3)
Requirement already satisfied: tornado>=4 in /root/anaconda3/lib/python3.6/site-packages (from notebook->jupyter->hyperas) (5.0.2)
Requirement already satisfied: ipython in /root/anaconda3/lib/python3.6/site-packages (from jupyter-console->jupyter->hyperas) (6.4.0)
Requirement already satisfied: prompt_toolkit<2.0.0,>=1.0.0 in /root/anaconda3/lib/python3.6/site-packages (from jupyter-console->jupyter->hyperas) (1.0.15)
Requirement already satisfied: widgetsnbextension~=3.2.0 in /root/anaconda3/lib/python3.6/site-packages (from ipywidgets->jupyter->hyperas) (3.2.1)
Requirement already satisfied: MarkupSafe>=0.23 in /root/anaconda3/lib/python3.6/site-packages (from jinja2->nbconvert->hyperas) (1.0)
Requirement already satisfied: html5lib!=1.0b1,!=1.0b2,!=1.0b3,!=1.0b4,!=1.0b5,!=1.0b6,!=1.0b7,!=1.0b8,>=0.99999999pre in /root/anaconda3/lib/python3.6/site-packages (from bleach->nbconvert->hyperas) (1.0.1)
Requirement already satisfied: python-dateutil>=2.1 in /root/anaconda3/lib/python3.6/site-packages (from jupyter-client>=5.2.0->notebook->jupyter->hyperas) (2.7.3)
Requirement already satisfied: pexpect; sys_platform != "win32" in /root/anaconda3/lib/python3.6/site-packages (from ipython->jupyter-console->jupyter->hyperas) (4.5.0)
Requirement already satisfied: pickleshare in /root/anaconda3/lib/python3.6/site-packages (from ipython->jupyter-console->jupyter->hyperas) (0.7.4)
Requirement already satisfied: backcall in /root/anaconda3/lib/python3.6/site-packages (from ipython->jupyter-console->jupyter->hyperas) (0.1.0)
Requirement already satisfied: setuptools>=18.5 in /root/.local/lib/python3.6/site-packages (from ipython->jupyter-console->jupyter->hyperas) (41.1.0)
Requirement already satisfied: jedi>=0.10 in /root/anaconda3/lib/python3.6/site-packages (from ipython->jupyter-console->jupyter->hyperas) (0.12.0)
Requirement already satisfied: simplegeneric>0.8 in /root/anaconda3/lib/python3.6/site-packages (from ipython->jupyter-console->jupyter->hyperas) (0.8.1)
Requirement already satisfied: wcwidth in /root/anaconda3/lib/python3.6/site-packages (from prompt_toolkit<2.0.0,>=1.0.0->jupyter-console->jupyter->hyperas) (0.1.7)
Requirement already satisfied: webencodings in /root/anaconda3/lib/python3.6/site-packages (from html5lib!=1.0b1,!=1.0b2,!=1.0b3,!=1.0b4,!=1.0b5,!=1.0b6,!=1.0b7,!=1.0b8,>=0.99999999pre->bleach->nbconvert->hyperas) (0.5.1)
Requirement already satisfied: ptyprocess>=0.5 in /root/anaconda3/lib/python3.6/site-packages (from pexpect; sys_platform != "win32"->ipython->jupyter-console->jupyter->hyperas) (0.5.2)
Requirement already satisfied: parso>=0.2.0 in /root/anaconda3/lib/python3.6/site-packages (from jedi>=0.10->ipython->jupyter-console->jupyter->hyperas) (0.2.0)
WARNING: You are using pip version 19.2.2, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

















In [2]:



    


# Basic compuational libaries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import seaborn as sns
%matplotlib inline

np.random.seed(2)

from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
import itertools
from sklearn.model_selection import KFold

from keras.utils.np_utils import to_categorical # convert to one-hot-encoding
from keras.layers import Dense, Dropout, Conv2D, GlobalAveragePooling2D, Flatten, GlobalMaxPooling2D
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization
from keras.models import Sequential
from keras.optimizers import RMSprop, Adam, SGD, Nadam
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import ReduceLROnPlateau, ModelCheckpoint, EarlyStopping
from keras import regularizers

# Import hyperopt for tunning hyper params
from hyperopt import hp, tpe, fmin
from hyperopt import space_eval

sns.set(style='white', context='notebook', palette='deep')
# Set the random seed
random_seed = 2



    


















    






/root/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/root/anaconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/root/anaconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/root/anaconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/root/anaconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/root/anaconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/root/anaconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])

2. Tiền xử lý

Đọc bộ dữ liệu MNIST, chúng ta chia bộ dữ liệu thành tập train/valid/test. Đồng thời chuẩn hóa dữ liệu về khoảng [0-1] để giúp tăng tốc quá trình hội tụ. Tập valid của chúng ta sẽ gồm 20% tập train.





In [3]:



    


def data():
    # Load the data
    train = pd.read_csv("../input/digit-recognizer/train.csv")
    test = pd.read_csv("../input/digit-recognizer/test.csv")
    Y_train = train["label"]
    
    # Drop 'label' column
    X_train = train.drop(labels = ["label"],axis = 1) 
    
    # Normalize the data
    X_train = X_train / 255.0
    test = test / 255.0
    
    # Reshape image in 3 dimensions (height = 28px, width = 28px , canal = 1)
    X_train = X_train.values.reshape(-1,28,28,1)
    test = test.values.reshape(-1,28,28,1)
    
    # Encode labels to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0])
    Y_train = to_categorical(Y_train, num_classes = 10)
    
    return X_train, Y_train, test

X, Y, X_test = data()

Kiểm tra phân bố của nhãn

Chúng ta thấy rằng số lượng mẫu dữ liệu cho mỗi nhãn tương đương nhau.





In [4]:



    


g = sns.countplot(np.argmax(Y, axis=1))

Thử nhìn qua một số mẫu trong tập huấn luyện. Chúng ta thấy rằng hầu hết các ảnh đều rõ nét và tương đối dễ dàng để nhận dạng.





In [5]:



    


for i in range(0, 9):
    plt.subplot(330 + (i+1))
    plt.imshow(X[i][:,:,0], cmap=plt.get_cmap('gray'))
    plt.title(np.argmax(Y[i]));
    plt.axis('off')
plt.tight_layout()

Định nghĩa số epochs cần huấn luyện và bachsize





In [6]:



    


epochs = 30 # Turn epochs to 30 to get 0.9967 accuracy
batch_size = 64

3. Data Augmentation

Kĩ thuật data augmentation được sử dụng để phát sinh thêm những mẫu dữ liệu mới bằng cách áp dụng các kĩ thuật xử lý ảnh trên bức ảnh. Các phép biến đổi nhỏ này phải đảm bảo không làm thay đổi nhãn của bức ảnh.

Một số kĩ thuật phổ biến của data augmentation như là:

  • Rotation: Xoay một góc nhỏ
  • Translation: Tính tiến
  • Brightness, Staturation: Thay đổi độ sáng, tương phản
  • Zoom: zoom to/nhỏ bức ảnh
  • Elastic Distortion: biến dạng bức ảnh
  • Flip: lật trái/phải/trên/dưới.

Ở dưới đây, chúng ta sẽ chọn xoay 1 góc trong 0-10 độ. Zoom ảnh 0.1 lần, tịnh tiến 0.1 lần mỗi chiều.





In [7]:



    


# With data augmentation to prevent overfitting (accuracy 0.99286)
train_aug = ImageDataGenerator(        
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        )

test_aug = ImageDataGenerator()

4. Xây dưng mô hình CNN

CNN bao gồm tập hợp các lớp cơ bản bao gồm: convolution layer + nonlinear layer, pooling layer, fully connected layer. Các lớp này liên kết với nhau theo một thứ tự nhất định. Thông thường, một ảnh sẽ được lan truyền qua tầng convolution layer + nonlinear layer đầu tiên, sau đó các giá trị tính toán được sẽ lan truyền qua pooling layer, bộ ba convolution layer + nonlinear layer + pooling layer có thể được lặp lại nhiều lần trong network. Và sau đó được lan truyền qua tầng fully connected layer và softmax để tính sác xuất ảnh đó chứa vật thế gì.

Định nghĩa mô hình

Chúng ta sử dụng Keras Sequential API để định nghĩa mô hình. Các layer được thêm vào rất dễ dàng và tương đối linh động. Đầu tiên chúng ta sử dụng layer Conv2D trên ảnh đầu vào. Conv2D bao gồm một tập các filters cần phải học. Mỗi filters sẽ trược qua toàn bộ bức ảnh để detect các đặt trưng trên bức ảnh đó.

Pooling layer là tầng quan trọng và thường đứng sau tầng Conv. Tầng này có chức năng giảm chiều của feature maps trước đó. Đối với max-pooling, tầng này chỉ đơn giản chọn giá trị lớn nhất trong vùng có kích thước pooling_size x pooling_size (thường là 2x2). Tầng pooling này được sử dụng để giảm chi phí tính toán và giảm được overfit của mô hình.

Đồng thời, Dropout cũng được sử dụng để hạn chế overfit. Dropout sẽ bỏ đi ngẫu nhiên các neuron bằng cách nhân với mask zeros, do đó, giúp mô hình học được những đặc trưng hữu ích. Dropout trong hầu hết các trường hợp đều giúp tăng độ chính xác và hạn chết overfit của mô hình.

Ở tầng cuối cùng, chúng ta flatten feature matrix thành một vector, sau đó sử dụng các tầng fully connected layers để phân loại ảnh thành các lớp cho trước.

Để giúp mô hình hội tụ gần với gobal minima chúng ta sử dụng annealing learning rate. Learning sẽ được điều chỉnh nhỏ dần sau mỗi lần cập nhật nếu như sau một số bước nhất định mà loss của mô hình không giảm nữa. Để giảm thời gian tính toán, chúng ta có thể sử dụng learning ban đầu lớn, sau đó giảm dần để mô hình hội tụ nhanh hơn.

Ngoài ra, chúng ta sử dụng early stopping để hạn chế hiện tượng overfit của mô hình. early stopping sẽ dừng quá trình huấn luyện nếu như loss trên tập validation tăng dần trong khi trên tập lại giảm.

Sử dụng hyperas để tunning siêu tham số

Trong quá trình định nghĩa mô hình, chúng ta sẽ lồng vào đó các đoạn mã để hỗ trợ quá trình search siêu tham số đã được định nghĩa ở trên. Chúng ta sẽ cần search các tham số như filter_size, pooling_size, dropout rate, dense size. Đồng thời chúng ta cũng thử việc điều chỉnh cả optimizer của mô hình.





In [8]:



    


# Set the CNN model 
def train_model(train_generator, valid_generator, params):    
    model = Sequential()

    model.add(Conv2D(filters = params['conv1'], kernel_size = params['kernel_size_1'], padding = 'Same', 
                     activation ='relu', input_shape = (28,28,1)))
    model.add(BatchNormalization())
    model.add(Conv2D(filters = params['conv2'], kernel_size = params['kernel_size_2'], padding = 'Same', 
                     activation ='relu'))
    model.add(MaxPool2D(pool_size = params['pooling_size_1']))
    model.add(Dropout(params['dropout1']))

    model.add(BatchNormalization())
    model.add(Conv2D(filters = params['conv3'], kernel_size = params['kernel_size_3'], padding = 'Same', 
                     activation ='relu'))
    model.add(BatchNormalization())
    model.add(Conv2D(filters = params['conv4'], kernel_size = params['kernel_size_4'], padding = 'Same', 
                     activation ='relu'))
    model.add(MaxPool2D(pool_size = params['pooling_size_1'], strides=(2,2)))
    model.add(Dropout(params['dropout2']))

    model.add(Flatten())
    model.add(BatchNormalization())
    model.add(Dense(params['dense1'], activation = "relu"))
    model.add(Dropout(params['dropout3']))
    model.add(Dense(10, activation = "softmax"))
    
    if params['opt'] == 'rmsprop':
        opt = RMSprop()
    elif params['opt'] == 'sgd':
        opt = SGD()
    elif params['opt'] == 'nadam':
        opt = Nadam()
    else:
        opt = Adam()
    
    model.compile(loss=params['loss'], optimizer=opt, metrics=['acc'])
        
    reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.3, patience=2, mode='auto', cooldown=2, min_lr=1e-7)
    early = EarlyStopping(monitor='val_loss', patience=3)
    
    callbacks_list = [reduce_lr, early]    
    
    history = model.fit_generator(train_generator,
                                  validation_data=valid_generator,
                                  steps_per_epoch=len(train_generator),
                                  validation_steps=len(valid_generator),
                                  callbacks=callbacks_list, epochs = epochs,
                                  verbose=2)
    
    score, acc = model.evaluate_generator(valid_generator, steps=len(valid_generator), verbose=0)
    
    return acc, model, history

5. Hyper-params tunning

Chúng ta sử dụng Hyperas để tunning các tham số. Hyperas sẽ phát sinh bộ tham số dựa trên khai báo ở trên. Sau đó huấn luyện mô hình và đánh giá trên tập validation. Bộ tham số có độ chính xác cao nhất trên tập validation sẽ được ghi nhận lại.

5.1 Khai báo không gian tìm kiếm siêu tham số

Có rất nhiều siêu tham số cần được tunning như: kiến trúc mạng, số filter, kích thước mỗi filters, kích thước pooling, các cách khởi tạo, hàm kích hoạt, tỉ lệ dropout,... Trong phần này, chúng ta sẽ tập trung vào các tham số như kích thước filter, số filters, pooling size.

Đầu tiên, chúng ta cần khai báo các siêu tham để hyperas có thể tìm kiếm trong tập đấy. Ở mỗi tầng conv, chúng ta sẽ tunning kích thước filter, filter size. Ở tầng pooling, kích thước pooling size sẽ được tunning. Đồng thời, tỉ lệ dropout ở tầng Dropout cũng được tunning. Số filters ở tầng conv thường từ 16 -> 1024, kích thước filter hay thường dùng nhất trong là 3 với 5. Còn tỉ lệ dropout nằm trong đoạn 0-1





In [9]:



    


#This is the space of hyperparameters that we will search
space = {
    'opt':hp.choice('opt', ['adam', 'sgd', 'rmsprop']),
    
    'conv1':hp.choice('conv1', [16, 32, 64, 128]),
    'conv2':hp.choice('conv2', [16, 32, 64, 128]),
    'kernel_size_1': hp.choice('kernel_size_1', [3, 5]),
    'kernel_size_2': hp.choice('kernel_size_2', [3, 5]),
    'dropout1': hp.choice('dropout1', [0, 0.25, 0.5]),
    'pooling_size_1': hp.choice('pooling_size_1', [2, 3]),
    
    'conv3':hp.choice('conv3', [32, 64, 128, 256, 512]),
    'conv4':hp.choice('conv4', [32, 64, 128, 256, 512]),
    'kernel_size_3': hp.choice('kernel_size_3', [3, 5]),
    'kernel_size_4': hp.choice('kernel_size_4', [3, 5]),
    'dropout2':hp.choice('dropout2', [0, 0.25, 0.5]),
    'pooling_size_2': hp.choice('pooling_size_2', [2, 3]),
    
    'dense1':hp.choice('dense1', [128, 256, 512, 1024]),
    'dropout3':hp.choice('dropout3', [0, 0.25, 0.5]),
    
    'loss': hp.choice('loss', ['categorical_crossentropy', 'kullback_leibler_divergence']),
}

5.2 Optimze để tìm bộ tham số tốt nhất

Hyperas sẽ phát sinh các bộ tham số giữ trên không gian tìm kiếm định nghĩa trước của chúng ta. Sau đó thư viện sẽ hỗ trợ quá trình tìm kiếm các tham số này đơn giản bằng một số API có sẵn.





In [10]:



    


X_train, X_val, Y_train, Y_val = train_test_split(X, Y, test_size = 0.2, random_state=random_seed)
# only apply data augmentation with train data
train_gen = train_aug.flow(X_train, Y_train, batch_size=batch_size)
valid_gen = test_aug.flow(X_val, Y_val, batch_size=batch_size)

def optimize(params):
    acc, model, history = train_model(train_gen, valid_gen, params)
    
    return -acc

Chạy quá trình search tham số. Bộ siêu tham số tốt nhất sẽ được ghi nhận lại để chúng ta sử dụng trong mô hình cuối cùng.





In [11]:



    


best = fmin(fn = optimize, space = space, 
            algo = tpe.suggest, max_evals = 50) # change to 50 to search more



    


















    






  0%|          | 0/50 [00:00<?, ?it/s, best loss: ?]









    






WARNING: Logging before flag parsing goes to stderr.

W0914 09:43:05.983868 140240089409280 deprecation_wrapper.py:119] From /root/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

W0914 09:43:06.065519 140240089409280 deprecation_wrapper.py:119] From /root/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W0914 09:43:06.075301 140240089409280 deprecation_wrapper.py:119] From /root/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

W0914 09:43:06.123427 140240089409280 deprecation_wrapper.py:119] From /root/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

W0914 09:43:06.125205 140240089409280 deprecation_wrapper.py:119] From /root/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0914 09:43:10.018944 140240089409280 deprecation_wrapper.py:119] From /root/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

W0914 09:43:10.144633 140240089409280 deprecation_wrapper.py:119] From /root/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3976: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

W0914 09:43:10.155966 140240089409280 deprecation.py:506] From /root/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
W0914 09:43:10.834075 140240089409280 deprecation_wrapper.py:119] From /root/anaconda3/lib/python3.6/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

W0914 09:43:10.981822 140240089409280 deprecation.py:323] From /root/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where










    






Epoch 1/30                                          
 - 19s - loss: 0.5554 - acc: 0.8958 - val_loss: 0.3073 - val_acc: 0.9707

Epoch 2/30                                          
 - 12s - loss: 0.1966 - acc: 0.9566 - val_loss: 0.1294 - val_acc: 0.9815

Epoch 3/30                                          
 - 13s - loss: 0.1257 - acc: 0.9689 - val_loss: 0.1337 - val_acc: 0.9817

Epoch 4/30                                          
 - 12s - loss: 0.1069 - acc: 0.9740 - val_loss: 0.0717 - val_acc: 0.9871

Epoch 5/30                                          
 - 13s - loss: 0.1014 - acc: 0.9768 - val_loss: 0.0616 - val_acc: 0.9875

Epoch 6/30                                          
 - 12s - loss: 0.0942 - acc: 0.9792 - val_loss: 0.0549 - val_acc: 0.9881

Epoch 7/30                                          
 - 13s - loss: 0.0926 - acc: 0.9790 - val_loss: 0.0696 - val_acc: 0.9896

Epoch 8/30                                          
 - 13s - loss: 0.0939 - acc: 0.9791 - val_loss: 0.0530 - val_acc: 0.9915

Epoch 9/30                                          
 - 13s - loss: 0.0891 - acc: 0.9810 - val_loss: 0.0581 - val_acc: 0.9917

Epoch 10/30                                         
 - 12s - loss: 0.0996 - acc: 0.9806 - val_loss: 0.0514 - val_acc: 0.9920

Epoch 11/30                                         
 - 13s - loss: 0.0953 - acc: 0.9818 - val_loss: 0.0505 - val_acc: 0.9923

Epoch 12/30                                         
 - 13s - loss: 0.0998 - acc: 0.9815 - val_loss: 0.0642 - val_acc: 0.9913

Epoch 13/30                                         
 - 13s - loss: 0.0931 - acc: 0.9825 - val_loss: 0.0482 - val_acc: 0.9926

Epoch 14/30                                         
 - 13s - loss: 0.0910 - acc: 0.9826 - val_loss: 0.0467 - val_acc: 0.9930

Epoch 15/30                                         
 - 13s - loss: 0.0881 - acc: 0.9832 - val_loss: 0.0739 - val_acc: 0.9920

Epoch 16/30                                         
 - 13s - loss: 0.0870 - acc: 0.9836 - val_loss: 0.0677 - val_acc: 0.9899

Epoch 17/30                                         
 - 13s - loss: 0.0607 - acc: 0.9890 - val_loss: 0.0434 - val_acc: 0.9943

Epoch 18/30                                         
 - 13s - loss: 0.0498 - acc: 0.9907 - val_loss: 0.0434 - val_acc: 0.9946

Epoch 19/30                                         
 - 13s - loss: 0.0503 - acc: 0.9908 - val_loss: 0.0501 - val_acc: 0.9937

Epoch 20/30                                         
 - 13s - loss: 0.0438 - acc: 0.9912 - val_loss: 0.0391 - val_acc: 0.9952

Epoch 21/30                                         
 - 12s - loss: 0.0429 - acc: 0.9918 - val_loss: 0.0370 - val_acc: 0.9954

Epoch 22/30                                         
 - 13s - loss: 0.0418 - acc: 0.9922 - val_loss: 0.0394 - val_acc: 0.9950

Epoch 23/30                                         
 - 13s - loss: 0.0435 - acc: 0.9910 - val_loss: 0.0393 - val_acc: 0.9944

Epoch 24/30                                         
 - 13s - loss: 0.0398 - acc: 0.9920 - val_loss: 0.0372 - val_acc: 0.9949

Epoch 1/30                                                                       
 - 17s - loss: 0.9664 - acc: 0.9156 - val_loss: 1.0357 - val_acc: 0.9275         

Epoch 2/30                                                                       
 - 13s - loss: 0.9532 - acc: 0.9346 - val_loss: 0.4416 - val_acc: 0.9712         

Epoch 3/30                                                                       
 - 13s - loss: 0.9019 - acc: 0.9403 - val_loss: 0.8849 - val_acc: 0.9435         

Epoch 4/30                                                                       
 - 13s - loss: 0.8870 - acc: 0.9428 - val_loss: 0.6445 - val_acc: 0.9587         

Epoch 5/30                                                                       
 - 13s - loss: 0.5074 - acc: 0.9675 - val_loss: 0.3391 - val_acc: 0.9785         

Epoch 6/30                                                                       
 - 13s - loss: 0.4676 - acc: 0.9701 - val_loss: 0.3290 - val_acc: 0.9786         

Epoch 7/30                                                                       
 - 13s - loss: 0.4251 - acc: 0.9728 - val_loss: 0.3726 - val_acc: 0.9763         

Epoch 8/30                                                                       
 - 13s - loss: 0.4235 - acc: 0.9729 - val_loss: 0.3774 - val_acc: 0.9758         

Epoch 9/30                                                                       
 - 13s - loss: 0.3975 - acc: 0.9743 - val_loss: 0.2364 - val_acc: 0.9848         

Epoch 10/30                                                                      
 - 13s - loss: 0.3164 - acc: 0.9796 - val_loss: 0.2160 - val_acc: 0.9863         

Epoch 11/30                                                                      
 - 12s - loss: 0.2887 - acc: 0.9811 - val_loss: 0.2184 - val_acc: 0.9857         

Epoch 12/30                                                                      
 - 13s - loss: 0.2726 - acc: 0.9821 - val_loss: 0.1987 - val_acc: 0.9870         

Epoch 13/30                                                                      
 - 13s - loss: 0.2596 - acc: 0.9831 - val_loss: 0.2118 - val_acc: 0.9862         

Epoch 14/30                                                                      
 - 13s - loss: 0.2464 - acc: 0.9840 - val_loss: 0.2148 - val_acc: 0.9863         

Epoch 15/30                                                                      
 - 13s - loss: 0.2507 - acc: 0.9839 - val_loss: 0.1660 - val_acc: 0.9893         

Epoch 16/30                                                                      
 - 13s - loss: 0.2186 - acc: 0.9856 - val_loss: 0.1904 - val_acc: 0.9880         

Epoch 17/30                                                                      
 - 13s - loss: 0.2297 - acc: 0.9850 - val_loss: 0.1671 - val_acc: 0.9895         

Epoch 18/30                                                                      
 - 13s - loss: 0.2243 - acc: 0.9854 - val_loss: 0.1535 - val_acc: 0.9899         

Epoch 19/30                                                                      
 - 13s - loss: 0.2341 - acc: 0.9850 - val_loss: 0.1759 - val_acc: 0.9885         

Epoch 20/30                                                                      
 - 13s - loss: 0.2328 - acc: 0.9850 - val_loss: 0.1490 - val_acc: 0.9902         

Epoch 21/30                                                                      
 - 13s - loss: 0.2095 - acc: 0.9863 - val_loss: 0.1746 - val_acc: 0.9886         

Epoch 22/30                                                                      
 - 13s - loss: 0.2244 - acc: 0.9855 - val_loss: 0.1477 - val_acc: 0.9904         

Epoch 23/30                                                                      
 - 13s - loss: 0.2187 - acc: 0.9859 - val_loss: 0.1455 - val_acc: 0.9904         

Epoch 24/30                                                                      
 - 13s - loss: 0.1993 - acc: 0.9871 - val_loss: 0.1623 - val_acc: 0.9896         

Epoch 25/30                                                                      
 - 13s - loss: 0.2076 - acc: 0.9865 - val_loss: 0.1603 - val_acc: 0.9893         

Epoch 26/30                                                                      
 - 13s - loss: 0.2081 - acc: 0.9866 - val_loss: 0.1567 - val_acc: 0.9900         

Epoch 1/30                                                                       
 - 16s - loss: 0.4960 - acc: 0.8443 - val_loss: 0.0618 - val_acc: 0.9818         

Epoch 2/30                                                                       
 - 13s - loss: 0.1609 - acc: 0.9529 - val_loss: 0.0576 - val_acc: 0.9845         

Epoch 3/30                                                                       
 - 13s - loss: 0.1358 - acc: 0.9620 - val_loss: 0.0406 - val_acc: 0.9893         

Epoch 4/30                                                                       
 - 13s - loss: 0.1236 - acc: 0.9657 - val_loss: 0.0579 - val_acc: 0.9869         

Epoch 5/30                                                                       
 - 13s - loss: 0.1175 - acc: 0.9705 - val_loss: 0.0731 - val_acc: 0.9875         

Epoch 6/30                                                                       
 - 13s - loss: 0.0870 - acc: 0.9771 - val_loss: 0.0348 - val_acc: 0.9919         

Epoch 7/30                                                                       
 - 13s - loss: 0.0813 - acc: 0.9798 - val_loss: 0.0335 - val_acc: 0.9927         

Epoch 8/30                                                                       
 - 13s - loss: 0.0744 - acc: 0.9815 - val_loss: 0.0335 - val_acc: 0.9923         

Epoch 9/30                                                                       
 - 13s - loss: 0.0745 - acc: 0.9807 - val_loss: 0.0292 - val_acc: 0.9930         

Epoch 10/30                                                                      
 - 13s - loss: 0.0695 - acc: 0.9824 - val_loss: 0.0333 - val_acc: 0.9920         

Epoch 11/30                                                                      
 - 13s - loss: 0.0708 - acc: 0.9822 - val_loss: 0.0388 - val_acc: 0.9919         

Epoch 12/30                                                                      
 - 13s - loss: 0.0655 - acc: 0.9843 - val_loss: 0.0257 - val_acc: 0.9938         

Epoch 13/30                                                                      
 - 13s - loss: 0.0644 - acc: 0.9843 - val_loss: 0.0278 - val_acc: 0.9936         

Epoch 14/30                                                                      
 - 13s - loss: 0.0645 - acc: 0.9845 - val_loss: 0.0263 - val_acc: 0.9942         

Epoch 15/30                                                                      
 - 13s - loss: 0.0578 - acc: 0.9854 - val_loss: 0.0307 - val_acc: 0.9930         

Epoch 1/30                                                                       
 - 17s - loss: 0.6574 - acc: 0.9177 - val_loss: 0.4988 - val_acc: 0.9615         

Epoch 2/30                                                                       
 - 13s - loss: 0.5163 - acc: 0.9549 - val_loss: 0.4268 - val_acc: 0.9658         

Epoch 3/30                                                                       
 - 13s - loss: 0.4795 - acc: 0.9609 - val_loss: 0.4681 - val_acc: 0.9636         

Epoch 4/30                                                                       
 - 13s - loss: 0.4307 - acc: 0.9666 - val_loss: 0.3215 - val_acc: 0.9758         

Epoch 5/30                                                                       
 - 13s - loss: 0.4220 - acc: 0.9670 - val_loss: 0.3196 - val_acc: 0.9764         

Epoch 6/30                                                                       
 - 13s - loss: 0.3804 - acc: 0.9719 - val_loss: 0.4115 - val_acc: 0.9706         

Epoch 7/30                                                                       
 - 13s - loss: 0.3977 - acc: 0.9716 - val_loss: 0.2873 - val_acc: 0.9792         

Epoch 8/30                                                                       
 - 13s - loss: 0.4015 - acc: 0.9712 - val_loss: 0.3924 - val_acc: 0.9729         

Epoch 9/30                                                                       
 - 13s - loss: 0.4319 - acc: 0.9701 - val_loss: 0.2615 - val_acc: 0.9820         

Epoch 10/30                                                                      
 - 13s - loss: 0.4116 - acc: 0.9716 - val_loss: 0.5734 - val_acc: 0.9612         

Epoch 11/30                                                                      
 - 13s - loss: 0.4008 - acc: 0.9729 - val_loss: 0.2799 - val_acc: 0.9815         

Epoch 12/30                                                                      
 - 13s - loss: 0.2759 - acc: 0.9809 - val_loss: 0.1867 - val_acc: 0.9855         

Epoch 13/30                                                                      
 - 13s - loss: 0.2192 - acc: 0.9846 - val_loss: 0.1530 - val_acc: 0.9898         

Epoch 14/30                                                                      
 - 13s - loss: 0.1964 - acc: 0.9864 - val_loss: 0.1338 - val_acc: 0.9905         

Epoch 15/30                                                                      
 - 14s - loss: 0.2035 - acc: 0.9860 - val_loss: 0.1199 - val_acc: 0.9917         

Epoch 16/30                                                                      
 - 13s - loss: 0.1799 - acc: 0.9877 - val_loss: 0.1060 - val_acc: 0.9927         

Epoch 17/30                                                                      
 - 13s - loss: 0.1713 - acc: 0.9882 - val_loss: 0.1342 - val_acc: 0.9908         

Epoch 18/30                                                                      
 - 13s - loss: 0.1708 - acc: 0.9881 - val_loss: 0.1145 - val_acc: 0.9918         

Epoch 19/30                                                                      
 - 13s - loss: 0.1475 - acc: 0.9896 - val_loss: 0.1034 - val_acc: 0.9927         

Epoch 20/30                                                                      
 - 13s - loss: 0.1479 - acc: 0.9896 - val_loss: 0.0757 - val_acc: 0.9943         

Epoch 21/30                                                                      
 - 13s - loss: 0.1301 - acc: 0.9908 - val_loss: 0.1102 - val_acc: 0.9926         

Epoch 22/30                                                                      
 - 14s - loss: 0.1316 - acc: 0.9911 - val_loss: 0.0869 - val_acc: 0.9936         

Epoch 23/30                                                                      
 - 13s - loss: 0.1329 - acc: 0.9905 - val_loss: 0.0835 - val_acc: 0.9943         

Epoch 1/30                                                                       
 - 16s - loss: 1.0239 - acc: 0.6619 - val_loss: 0.1741 - val_acc: 0.9469         

Epoch 2/30                                                                       
 - 13s - loss: 0.3178 - acc: 0.9034 - val_loss: 0.1258 - val_acc: 0.9625         

Epoch 3/30                                                                       
 - 13s - loss: 0.2087 - acc: 0.9358 - val_loss: 0.0757 - val_acc: 0.9760         

Epoch 4/30                                                                       
 - 13s - loss: 0.1657 - acc: 0.9510 - val_loss: 0.0638 - val_acc: 0.9807         

Epoch 5/30                                                                       
 - 13s - loss: 0.1426 - acc: 0.9560 - val_loss: 0.0464 - val_acc: 0.9846         

Epoch 6/30                                                                       
 - 13s - loss: 0.1204 - acc: 0.9629 - val_loss: 0.0570 - val_acc: 0.9827         

Epoch 7/30                                                                       
 - 13s - loss: 0.1091 - acc: 0.9672 - val_loss: 0.0400 - val_acc: 0.9875         

Epoch 8/30                                                                       
 - 13s - loss: 0.0976 - acc: 0.9707 - val_loss: 0.0722 - val_acc: 0.9795         

Epoch 9/30                                                                       
 - 13s - loss: 0.0908 - acc: 0.9719 - val_loss: 0.0343 - val_acc: 0.9893         

Epoch 10/30                                                                      
 - 13s - loss: 0.0882 - acc: 0.9729 - val_loss: 0.0390 - val_acc: 0.9892         

Epoch 11/30                                                                      
 - 13s - loss: 0.0828 - acc: 0.9738 - val_loss: 0.0303 - val_acc: 0.9899         

Epoch 12/30                                                                      
 - 13s - loss: 0.0747 - acc: 0.9774 - val_loss: 0.0358 - val_acc: 0.9885         

Epoch 13/30                                                                      
 - 13s - loss: 0.0715 - acc: 0.9781 - val_loss: 0.0246 - val_acc: 0.9927         

Epoch 14/30                                                                      
 - 13s - loss: 0.0714 - acc: 0.9785 - val_loss: 0.0246 - val_acc: 0.9930         

Epoch 15/30                                                                      
 - 13s - loss: 0.0692 - acc: 0.9789 - val_loss: 0.0283 - val_acc: 0.9902         

Epoch 16/30                                                                      
 - 13s - loss: 0.0624 - acc: 0.9810 - val_loss: 0.0243 - val_acc: 0.9929         

Epoch 17/30                                                                      
 - 13s - loss: 0.0549 - acc: 0.9827 - val_loss: 0.0229 - val_acc: 0.9923         

Epoch 18/30                                                                      
 - 13s - loss: 0.0588 - acc: 0.9823 - val_loss: 0.0220 - val_acc: 0.9924         

Epoch 19/30                                                                      
 - 14s - loss: 0.0565 - acc: 0.9834 - val_loss: 0.0212 - val_acc: 0.9932         

Epoch 20/30                                                                      
 - 13s - loss: 0.0563 - acc: 0.9834 - val_loss: 0.0254 - val_acc: 0.9917         

Epoch 21/30                                                                      
 - 13s - loss: 0.0544 - acc: 0.9833 - val_loss: 0.0199 - val_acc: 0.9939         

Epoch 22/30                                                                      
 - 13s - loss: 0.0542 - acc: 0.9838 - val_loss: 0.0233 - val_acc: 0.9921         

Epoch 23/30                                                                      
 - 13s - loss: 0.0541 - acc: 0.9843 - val_loss: 0.0231 - val_acc: 0.9918         

Epoch 24/30                                                                      
 - 13s - loss: 0.0509 - acc: 0.9843 - val_loss: 0.0206 - val_acc: 0.9935         

Epoch 1/30                                                                       
 - 21s - loss: 1.6756 - acc: 0.8689 - val_loss: 0.6940 - val_acc: 0.9527         

Epoch 2/30                                                                       
 - 16s - loss: 0.9866 - acc: 0.9319 - val_loss: 0.9049 - val_acc: 0.9394         

Epoch 3/30                                                                       
 - 16s - loss: 0.8665 - acc: 0.9422 - val_loss: 0.5904 - val_acc: 0.9614         

Epoch 4/30                                                                       
 - 16s - loss: 0.7071 - acc: 0.9529 - val_loss: 0.5167 - val_acc: 0.9655         

Epoch 5/30                                                                       
 - 16s - loss: 0.6586 - acc: 0.9566 - val_loss: 0.4366 - val_acc: 0.9713         

Epoch 6/30                                                                       
 - 16s - loss: 0.6200 - acc: 0.9591 - val_loss: 0.7137 - val_acc: 0.9533         

Epoch 7/30                                                                       
 - 16s - loss: 0.6694 - acc: 0.9563 - val_loss: 0.6443 - val_acc: 0.9577         

Epoch 8/30                                                                       
 - 16s - loss: 0.4908 - acc: 0.9680 - val_loss: 0.2523 - val_acc: 0.9833         

Epoch 9/30                                                                       
 - 16s - loss: 0.3721 - acc: 0.9758 - val_loss: 0.2367 - val_acc: 0.9845         

Epoch 10/30                                                                      
 - 16s - loss: 0.3498 - acc: 0.9773 - val_loss: 0.1804 - val_acc: 0.9886         

Epoch 11/30                                                                      
 - 16s - loss: 0.3272 - acc: 0.9785 - val_loss: 0.2229 - val_acc: 0.9852         

Epoch 12/30                                                                      
 - 16s - loss: 0.2959 - acc: 0.9803 - val_loss: 0.1800 - val_acc: 0.9883         

Epoch 13/30                                                                      
 - 16s - loss: 0.2873 - acc: 0.9805 - val_loss: 0.1722 - val_acc: 0.9888         

Epoch 14/30                                                                      
 - 16s - loss: 0.2762 - acc: 0.9816 - val_loss: 0.1621 - val_acc: 0.9894         

Epoch 15/30                                                                      
 - 17s - loss: 0.2464 - acc: 0.9833 - val_loss: 0.1692 - val_acc: 0.9889         

Epoch 16/30                                                                      
 - 16s - loss: 0.2348 - acc: 0.9846 - val_loss: 0.1621 - val_acc: 0.9893         

Epoch 17/30                                                                      
 - 17s - loss: 0.2236 - acc: 0.9852 - val_loss: 0.1422 - val_acc: 0.9899         

Epoch 18/30                                                                      
 - 16s - loss: 0.2178 - acc: 0.9859 - val_loss: 0.1475 - val_acc: 0.9902         

Epoch 19/30                                                                      
 - 16s - loss: 0.2072 - acc: 0.9863 - val_loss: 0.1584 - val_acc: 0.9893         

Epoch 20/30                                                                      
 - 16s - loss: 0.2196 - acc: 0.9855 - val_loss: 0.1343 - val_acc: 0.9915         

Epoch 21/30                                                                      
 - 16s - loss: 0.2013 - acc: 0.9865 - val_loss: 0.1432 - val_acc: 0.9905         

Epoch 22/30                                                                      
 - 16s - loss: 0.2084 - acc: 0.9862 - val_loss: 0.1349 - val_acc: 0.9914         

Epoch 23/30                                                                      
 - 16s - loss: 0.1891 - acc: 0.9876 - val_loss: 0.1285 - val_acc: 0.9919         

Epoch 24/30                                                                      
 - 16s - loss: 0.1852 - acc: 0.9876 - val_loss: 0.1296 - val_acc: 0.9914         

Epoch 25/30                                                                      
 - 16s - loss: 0.1986 - acc: 0.9869 - val_loss: 0.1462 - val_acc: 0.9904         

Epoch 26/30                                                                      
 - 16s - loss: 0.1874 - acc: 0.9875 - val_loss: 0.1469 - val_acc: 0.9902         

Epoch 1/30                                                                       
 - 18s - loss: 0.4874 - acc: 0.8507 - val_loss: 0.0734 - val_acc: 0.9785         

Epoch 2/30                                                                       
 - 14s - loss: 0.1627 - acc: 0.9520 - val_loss: 0.0551 - val_acc: 0.9848         

Epoch 3/30                                                                       
 - 13s - loss: 0.1213 - acc: 0.9635 - val_loss: 0.0511 - val_acc: 0.9852         

Epoch 4/30                                                                       
 - 13s - loss: 0.1058 - acc: 0.9686 - val_loss: 0.0297 - val_acc: 0.9915         

Epoch 5/30                                                                       
 - 13s - loss: 0.0947 - acc: 0.9731 - val_loss: 0.0390 - val_acc: 0.9886         

Epoch 6/30                                                                       
 - 13s - loss: 0.0860 - acc: 0.9751 - val_loss: 0.0375 - val_acc: 0.9899         

Epoch 7/30                                                                       
 - 14s - loss: 0.0648 - acc: 0.9811 - val_loss: 0.0277 - val_acc: 0.9926         

Epoch 8/30                                                                       
 - 13s - loss: 0.0540 - acc: 0.9843 - val_loss: 0.0250 - val_acc: 0.9925         

Epoch 9/30                                                                       
 - 13s - loss: 0.0470 - acc: 0.9855 - val_loss: 0.0227 - val_acc: 0.9936         

Epoch 10/30                                                                      
 - 14s - loss: 0.0455 - acc: 0.9862 - val_loss: 0.0196 - val_acc: 0.9945         

Epoch 11/30                                                                      
 - 13s - loss: 0.0438 - acc: 0.9861 - val_loss: 0.0210 - val_acc: 0.9940         

Epoch 12/30                                                                      
 - 13s - loss: 0.0485 - acc: 0.9857 - val_loss: 0.0204 - val_acc: 0.9943         

Epoch 13/30                                                                      
 - 13s - loss: 0.0386 - acc: 0.9881 - val_loss: 0.0179 - val_acc: 0.9950         

Epoch 14/30                                                                      
 - 13s - loss: 0.0382 - acc: 0.9888 - val_loss: 0.0175 - val_acc: 0.9949         

Epoch 15/30                                                                      
 - 13s - loss: 0.0364 - acc: 0.9882 - val_loss: 0.0174 - val_acc: 0.9948         

Epoch 16/30                                                                      
 - 14s - loss: 0.0341 - acc: 0.9901 - val_loss: 0.0169 - val_acc: 0.9949         

Epoch 17/30                                                                      
 - 13s - loss: 0.0316 - acc: 0.9904 - val_loss: 0.0167 - val_acc: 0.9950         

Epoch 18/30                                                                      
 - 13s - loss: 0.0334 - acc: 0.9894 - val_loss: 0.0177 - val_acc: 0.9946         

Epoch 19/30                                                                      
 - 13s - loss: 0.0315 - acc: 0.9901 - val_loss: 0.0156 - val_acc: 0.9956         

Epoch 20/30                                                                      
 - 13s - loss: 0.0322 - acc: 0.9901 - val_loss: 0.0156 - val_acc: 0.9955         

Epoch 21/30                                                                      
 - 14s - loss: 0.0317 - acc: 0.9900 - val_loss: 0.0161 - val_acc: 0.9951         

Epoch 22/30                                                                      
 - 13s - loss: 0.0296 - acc: 0.9907 - val_loss: 0.0158 - val_acc: 0.9952         

Epoch 1/30                                                                       
 - 18s - loss: 0.3462 - acc: 0.9129 - val_loss: 0.1458 - val_acc: 0.9729

Epoch 2/30                                                          
 - 13s - loss: 0.1845 - acc: 0.9618 - val_loss: 0.0830 - val_acc: 0.9860

Epoch 3/30                                                          
 - 13s - loss: 0.1356 - acc: 0.9715 - val_loss: 0.1129 - val_acc: 0.9831

Epoch 4/30                                                          
 - 13s - loss: 0.1069 - acc: 0.9760 - val_loss: 0.0834 - val_acc: 0.9871

Epoch 5/30                                                          
 - 13s - loss: 0.0581 - acc: 0.9861 - val_loss: 0.0468 - val_acc: 0.9913

Epoch 6/30                                                          
 - 14s - loss: 0.0424 - acc: 0.9898 - val_loss: 0.0324 - val_acc: 0.9935

Epoch 7/30                                                          
 - 13s - loss: 0.0442 - acc: 0.9894 - val_loss: 0.0333 - val_acc: 0.9924

Epoch 8/30                                                          
 - 13s - loss: 0.0409 - acc: 0.9897 - val_loss: 0.0352 - val_acc: 0.9931

Epoch 9/30                                                          
 - 13s - loss: 0.0290 - acc: 0.9924 - val_loss: 0.0237 - val_acc: 0.9957

Epoch 10/30                                                         
 - 13s - loss: 0.0288 - acc: 0.9921 - val_loss: 0.0321 - val_acc: 0.9939

Epoch 11/30                                                         
 - 13s - loss: 0.0274 - acc: 0.9929 - val_loss: 0.0220 - val_acc: 0.9950

Epoch 12/30                                                         
 - 13s - loss: 0.0283 - acc: 0.9926 - val_loss: 0.0276 - val_acc: 0.9939

Epoch 13/30                                                         
 - 14s - loss: 0.0256 - acc: 0.9936 - val_loss: 0.0314 - val_acc: 0.9938

Epoch 14/30                                                         
 - 13s - loss: 0.0232 - acc: 0.9933 - val_loss: 0.0233 - val_acc: 0.9950

Epoch 1/30                                                          
 - 18s - loss: 0.3839 - acc: 0.8764 - val_loss: 0.0678 - val_acc: 0.9785

Epoch 2/30                                                          
 - 14s - loss: 0.1337 - acc: 0.9584 - val_loss: 0.0516 - val_acc: 0.9854

Epoch 3/30                                                          
 - 14s - loss: 0.0955 - acc: 0.9709 - val_loss: 0.0440 - val_acc: 0.9860

Epoch 4/30                                                          
 - 14s - loss: 0.0857 - acc: 0.9727 - val_loss: 0.0462 - val_acc: 0.9871

Epoch 5/30                                                          
 - 13s - loss: 0.0698 - acc: 0.9782 - val_loss: 0.0304 - val_acc: 0.9906

Epoch 6/30                                                          
 - 13s - loss: 0.0649 - acc: 0.9798 - val_loss: 0.0359 - val_acc: 0.9883

Epoch 7/30                                                          
 - 13s - loss: 0.0567 - acc: 0.9825 - val_loss: 0.0320 - val_acc: 0.9907

Epoch 8/30                                                          
 - 13s - loss: 0.0552 - acc: 0.9828 - val_loss: 0.0288 - val_acc: 0.9917

Epoch 9/30                                                          
 - 13s - loss: 0.0521 - acc: 0.9838 - val_loss: 0.0326 - val_acc: 0.9902

Epoch 10/30                                                         
 - 14s - loss: 0.0494 - acc: 0.9841 - val_loss: 0.0233 - val_acc: 0.9918

Epoch 11/30                                                         
 - 14s - loss: 0.0481 - acc: 0.9846 - val_loss: 0.0287 - val_acc: 0.9906

Epoch 12/30                                                         
 - 13s - loss: 0.0479 - acc: 0.9842 - val_loss: 0.0230 - val_acc: 0.9927

Epoch 13/30                                                         
 - 14s - loss: 0.0464 - acc: 0.9851 - val_loss: 0.0315 - val_acc: 0.9906

Epoch 14/30                                                         
 - 13s - loss: 0.0437 - acc: 0.9864 - val_loss: 0.0236 - val_acc: 0.9930

Epoch 15/30                                                         
 - 13s - loss: 0.0456 - acc: 0.9852 - val_loss: 0.0273 - val_acc: 0.9910

Epoch 1/30                                                          
 - 19s - loss: 0.6171 - acc: 0.7981 - val_loss: 0.1144 - val_acc: 0.9661

Epoch 2/30                                                          
 - 13s - loss: 0.2044 - acc: 0.9350 - val_loss: 0.0610 - val_acc: 0.9804

Epoch 3/30                                                          
 - 14s - loss: 0.1512 - acc: 0.9518 - val_loss: 0.0584 - val_acc: 0.9808

Epoch 4/30                                                          
 - 13s - loss: 0.1247 - acc: 0.9606 - val_loss: 0.0534 - val_acc: 0.9819

Epoch 5/30                                                          
 - 13s - loss: 0.1093 - acc: 0.9651 - val_loss: 0.0426 - val_acc: 0.9857

Epoch 6/30                                                          
 - 13s - loss: 0.0970 - acc: 0.9692 - val_loss: 0.0425 - val_acc: 0.9870

Epoch 7/30                                                          
 - 13s - loss: 0.0880 - acc: 0.9725 - val_loss: 0.0396 - val_acc: 0.9879

Epoch 8/30                                                          
 - 13s - loss: 0.0840 - acc: 0.9735 - val_loss: 0.0314 - val_acc: 0.9896

Epoch 9/30                                                          
 - 14s - loss: 0.0759 - acc: 0.9761 - val_loss: 0.0391 - val_acc: 0.9874

Epoch 10/30                                                         
 - 14s - loss: 0.0720 - acc: 0.9776 - val_loss: 0.0279 - val_acc: 0.9901

Epoch 11/30                                                         
 - 13s - loss: 0.0696 - acc: 0.9781 - val_loss: 0.0369 - val_acc: 0.9905

Epoch 12/30                                                         
 - 13s - loss: 0.0696 - acc: 0.9785 - val_loss: 0.0271 - val_acc: 0.9918

Epoch 13/30                                                         
 - 13s - loss: 0.0616 - acc: 0.9807 - val_loss: 0.0255 - val_acc: 0.9923

Epoch 14/30                                                         
 - 13s - loss: 0.0615 - acc: 0.9802 - val_loss: 0.0263 - val_acc: 0.9915

Epoch 15/30                                                         
 - 13s - loss: 0.0568 - acc: 0.9817 - val_loss: 0.0287 - val_acc: 0.9917

Epoch 16/30                                                         
 - 13s - loss: 0.0584 - acc: 0.9819 - val_loss: 0.0257 - val_acc: 0.9920

Epoch 1/30                                                           
 - 20s - loss: 0.4689 - acc: 0.8664 - val_loss: 0.1439 - val_acc: 0.9642

Epoch 2/30                                                           
 - 14s - loss: 0.1875 - acc: 0.9494 - val_loss: 0.0653 - val_acc: 0.9775

Epoch 3/30                                                           
 - 13s - loss: 0.1474 - acc: 0.9587 - val_loss: 0.0494 - val_acc: 0.9864

Epoch 4/30                                                           
 - 14s - loss: 0.1163 - acc: 0.9664 - val_loss: 0.0449 - val_acc: 0.9868

Epoch 5/30                                                           
 - 14s - loss: 0.1085 - acc: 0.9690 - val_loss: 0.0532 - val_acc: 0.9832

Epoch 6/30                                                           
 - 14s - loss: 0.0968 - acc: 0.9724 - val_loss: 0.0390 - val_acc: 0.9882

Epoch 7/30                                                           
 - 14s - loss: 0.0925 - acc: 0.9725 - val_loss: 0.0324 - val_acc: 0.9894

Epoch 8/30                                                           
 - 13s - loss: 0.0977 - acc: 0.9727 - val_loss: 0.0340 - val_acc: 0.9888

Epoch 9/30                                                           
 - 14s - loss: 0.0794 - acc: 0.9767 - val_loss: 0.0294 - val_acc: 0.9911

Epoch 10/30                                                          
 - 14s - loss: 0.0838 - acc: 0.9761 - val_loss: 0.0540 - val_acc: 0.9869

Epoch 11/30                                                          
 - 13s - loss: 0.0861 - acc: 0.9764 - val_loss: 0.0399 - val_acc: 0.9877

Epoch 12/30                                                          
 - 14s - loss: 0.0607 - acc: 0.9826 - val_loss: 0.0260 - val_acc: 0.9929

Epoch 13/30                                                          
 - 14s - loss: 0.0481 - acc: 0.9857 - val_loss: 0.0257 - val_acc: 0.9930

Epoch 14/30                                                          
 - 14s - loss: 0.0441 - acc: 0.9870 - val_loss: 0.0216 - val_acc: 0.9935

Epoch 15/30                                                          
 - 13s - loss: 0.0450 - acc: 0.9864 - val_loss: 0.0249 - val_acc: 0.9933

Epoch 16/30                                                          
 - 14s - loss: 0.0427 - acc: 0.9875 - val_loss: 0.0283 - val_acc: 0.9921

Epoch 17/30                                                          
 - 14s - loss: 0.0367 - acc: 0.9886 - val_loss: 0.0209 - val_acc: 0.9939

Epoch 18/30                                                          
 - 14s - loss: 0.0334 - acc: 0.9902 - val_loss: 0.0223 - val_acc: 0.9937

Epoch 19/30                                                          
 - 14s - loss: 0.0329 - acc: 0.9901 - val_loss: 0.0224 - val_acc: 0.9948

Epoch 20/30                                                          
 - 14s - loss: 0.0328 - acc: 0.9903 - val_loss: 0.0203 - val_acc: 0.9942

Epoch 21/30                                                          
 - 14s - loss: 0.0333 - acc: 0.9900 - val_loss: 0.0223 - val_acc: 0.9943

Epoch 22/30                                                          
 - 14s - loss: 0.0297 - acc: 0.9911 - val_loss: 0.0177 - val_acc: 0.9946

Epoch 23/30                                                          
 - 14s - loss: 0.0312 - acc: 0.9912 - val_loss: 0.0224 - val_acc: 0.9944

Epoch 24/30                                                          
 - 13s - loss: 0.0291 - acc: 0.9914 - val_loss: 0.0213 - val_acc: 0.9943

Epoch 25/30                                                          
 - 14s - loss: 0.0279 - acc: 0.9909 - val_loss: 0.0183 - val_acc: 0.9948

Epoch 1/30                                                           
 - 21s - loss: 0.2754 - acc: 0.9363 - val_loss: 0.2048 - val_acc: 0.9676

Epoch 2/30                                                           
 - 14s - loss: 0.1504 - acc: 0.9709 - val_loss: 0.0826 - val_acc: 0.9850

Epoch 3/30                                                           
 - 14s - loss: 0.1017 - acc: 0.9784 - val_loss: 0.0567 - val_acc: 0.9879

Epoch 4/30                                                           
 - 14s - loss: 0.0885 - acc: 0.9801 - val_loss: 0.0714 - val_acc: 0.9887

Epoch 5/30                                                           
 - 14s - loss: 0.0720 - acc: 0.9837 - val_loss: 0.0469 - val_acc: 0.9913

Epoch 6/30                                                           
 - 14s - loss: 0.0533 - acc: 0.9864 - val_loss: 0.0590 - val_acc: 0.9904

Epoch 7/30                                                           
 - 14s - loss: 0.0631 - acc: 0.9846 - val_loss: 0.0561 - val_acc: 0.9894

Epoch 8/30                                                           
 - 14s - loss: 0.0353 - acc: 0.9910 - val_loss: 0.0250 - val_acc: 0.9939

Epoch 9/30                                                           
 - 14s - loss: 0.0225 - acc: 0.9934 - val_loss: 0.0235 - val_acc: 0.9948

Epoch 10/30                                                          
 - 14s - loss: 0.0220 - acc: 0.9938 - val_loss: 0.0243 - val_acc: 0.9936

Epoch 11/30                                                          
 - 14s - loss: 0.0184 - acc: 0.9943 - val_loss: 0.0210 - val_acc: 0.9946

Epoch 12/30                                                          
 - 14s - loss: 0.0161 - acc: 0.9952 - val_loss: 0.0205 - val_acc: 0.9950

Epoch 13/30                                                          
 - 14s - loss: 0.0192 - acc: 0.9942 - val_loss: 0.0242 - val_acc: 0.9935

Epoch 14/30                                                          
 - 14s - loss: 0.0190 - acc: 0.9943 - val_loss: 0.0245 - val_acc: 0.9945

Epoch 15/30                                                          
 - 14s - loss: 0.0137 - acc: 0.9960 - val_loss: 0.0218 - val_acc: 0.9954

Epoch 1/30                                                                        
 - 22s - loss: 0.5127 - acc: 0.8815 - val_loss: 0.3727 - val_acc: 0.9604          

Epoch 2/30                                                                        
 - 15s - loss: 0.2329 - acc: 0.9503 - val_loss: 0.1801 - val_acc: 0.9793          

Epoch 3/30                                                                        
 - 14s - loss: 0.1848 - acc: 0.9592 - val_loss: 0.1309 - val_acc: 0.9751          

Epoch 4/30                                                                        
 - 14s - loss: 0.1597 - acc: 0.9665 - val_loss: 0.1286 - val_acc: 0.9854          

Epoch 5/30                                                                        
 - 14s - loss: 0.1513 - acc: 0.9687 - val_loss: 0.1181 - val_acc: 0.9815          

Epoch 6/30                                                                        
 - 14s - loss: 0.1572 - acc: 0.9696 - val_loss: 0.0928 - val_acc: 0.9892          

Epoch 7/30                                                                        
 - 14s - loss: 0.1491 - acc: 0.9701 - val_loss: 0.1040 - val_acc: 0.9886          

Epoch 8/30                                                                        
 - 14s - loss: 0.1456 - acc: 0.9723 - val_loss: 0.0903 - val_acc: 0.9904          

Epoch 9/30                                                                        
 - 14s - loss: 0.1502 - acc: 0.9740 - val_loss: 0.0797 - val_acc: 0.9887          

Epoch 10/30                                                                       
 - 14s - loss: 0.1533 - acc: 0.9745 - val_loss: 0.0630 - val_acc: 0.9923          

Epoch 11/30                                                                       
 - 14s - loss: 0.1473 - acc: 0.9753 - val_loss: 0.1024 - val_acc: 0.9867          

Epoch 12/30                                                                         
 - 14s - loss: 0.1444 - acc: 0.9773 - val_loss: 0.0659 - val_acc: 0.9910            

Epoch 13/30                                                                         
 - 15s - loss: 0.1214 - acc: 0.9816 - val_loss: 0.0496 - val_acc: 0.9939            

Epoch 14/30                                                                         
 - 14s - loss: 0.1000 - acc: 0.9836 - val_loss: 0.0544 - val_acc: 0.9937            

Epoch 15/30                                                                         
 - 14s - loss: 0.0914 - acc: 0.9850 - val_loss: 0.0555 - val_acc: 0.9942            

Epoch 16/30                                                                         
 - 14s - loss: 0.0852 - acc: 0.9862 - val_loss: 0.0532 - val_acc: 0.9943            

Epoch 1/30                                                                          
 - 23s - loss: 0.2821 - acc: 0.9110 - val_loss: 0.0573 - val_acc: 0.9829            

Epoch 2/30                                                                          
 - 15s - loss: 0.1029 - acc: 0.9681 - val_loss: 0.0449 - val_acc: 0.9864            

Epoch 3/30                                                                          
 - 15s - loss: 0.0820 - acc: 0.9738 - val_loss: 0.0434 - val_acc: 0.9875            

Epoch 4/30                                                                          
 - 15s - loss: 0.0646 - acc: 0.9794 - val_loss: 0.0297 - val_acc: 0.9911            

Epoch 5/30                                                                          
 - 15s - loss: 0.0583 - acc: 0.9808 - val_loss: 0.0317 - val_acc: 0.9895            

Epoch 6/30                                                                          
 - 15s - loss: 0.0523 - acc: 0.9835 - val_loss: 0.0316 - val_acc: 0.9892            

Epoch 7/30                                                                          
 - 15s - loss: 0.0454 - acc: 0.9856 - val_loss: 0.0255 - val_acc: 0.9915            

Epoch 8/30                                                                          
 - 15s - loss: 0.0411 - acc: 0.9875 - val_loss: 0.0258 - val_acc: 0.9923            

Epoch 9/30                                                                          
 - 15s - loss: 0.0403 - acc: 0.9874 - val_loss: 0.0236 - val_acc: 0.9919            

Epoch 10/30                                                                         
 - 15s - loss: 0.0389 - acc: 0.9885 - val_loss: 0.0235 - val_acc: 0.9926            

Epoch 11/30                                                                         
 - 15s - loss: 0.0398 - acc: 0.9875 - val_loss: 0.0253 - val_acc: 0.9917            

Epoch 12/30                                                                         
 - 15s - loss: 0.0369 - acc: 0.9885 - val_loss: 0.0241 - val_acc: 0.9924            

Epoch 13/30                                                                         
 - 15s - loss: 0.0343 - acc: 0.9890 - val_loss: 0.0221 - val_acc: 0.9927            

Epoch 14/30                                                                         
 - 15s - loss: 0.0374 - acc: 0.9882 - val_loss: 0.0223 - val_acc: 0.9927            

Epoch 15/30                                                                         
 - 15s - loss: 0.0373 - acc: 0.9885 - val_loss: 0.0216 - val_acc: 0.9935            

Epoch 16/30                                                                         
 - 15s - loss: 0.0343 - acc: 0.9896 - val_loss: 0.0230 - val_acc: 0.9927            

Epoch 17/30                                                                         
 - 15s - loss: 0.0354 - acc: 0.9885 - val_loss: 0.0234 - val_acc: 0.9921            

Epoch 18/30                                                                         
 - 15s - loss: 0.0331 - acc: 0.9893 - val_loss: 0.0208 - val_acc: 0.9932            

Epoch 19/30                                                                         
 - 15s - loss: 0.0372 - acc: 0.9885 - val_loss: 0.0216 - val_acc: 0.9930            

Epoch 20/30                                                                         
 - 15s - loss: 0.0348 - acc: 0.9892 - val_loss: 0.0246 - val_acc: 0.9924            

Epoch 21/30                                                                         
 - 15s - loss: 0.0339 - acc: 0.9897 - val_loss: 0.0187 - val_acc: 0.9933            

Epoch 22/30                                                                         
 - 15s - loss: 0.0334 - acc: 0.9889 - val_loss: 0.0232 - val_acc: 0.9929            

Epoch 23/30                                                                         
 - 15s - loss: 0.0348 - acc: 0.9891 - val_loss: 0.0198 - val_acc: 0.9936            

Epoch 24/30                                                                         
 - 15s - loss: 0.0333 - acc: 0.9894 - val_loss: 0.0234 - val_acc: 0.9925            

Epoch 1/30                                                                          
 - 27s - loss: 0.5423 - acc: 0.8826 - val_loss: 0.3249 - val_acc: 0.9565            

Epoch 2/30                                                                          
 - 18s - loss: 0.2459 - acc: 0.9502 - val_loss: 0.1190 - val_acc: 0.9810            

Epoch 3/30                                                                          
 - 18s - loss: 0.1682 - acc: 0.9612 - val_loss: 0.0814 - val_acc: 0.9854            

Epoch 4/30                                                                          
 - 18s - loss: 0.1362 - acc: 0.9669 - val_loss: 0.0607 - val_acc: 0.9879            

Epoch 5/30                                                                          
 - 18s - loss: 0.1351 - acc: 0.9674 - val_loss: 0.0900 - val_acc: 0.9830            

Epoch 6/30                                                                          
 - 18s - loss: 0.1263 - acc: 0.9684 - val_loss: 0.1372 - val_acc: 0.9761            

Epoch 7/30                                                                          
 - 18s - loss: 0.0803 - acc: 0.9800 - val_loss: 0.0233 - val_acc: 0.9933            

Epoch 8/30                                                                          
 - 18s - loss: 0.0620 - acc: 0.9839 - val_loss: 0.0246 - val_acc: 0.9949            

Epoch 9/30                                                                          
 - 18s - loss: 0.0544 - acc: 0.9857 - val_loss: 0.0291 - val_acc: 0.9933            

Epoch 10/30                                                                         
 - 18s - loss: 0.0541 - acc: 0.9865 - val_loss: 0.0289 - val_acc: 0.9931            

Epoch 1/30                                                                          
 - 23s - loss: 0.3664 - acc: 0.9165 - val_loss: 0.2863 - val_acc: 0.9715            

Epoch 2/30                                                                          
 - 14s - loss: 0.1919 - acc: 0.9632 - val_loss: 0.1237 - val_acc: 0.9845            

Epoch 3/30                                                                          
 - 14s - loss: 0.1414 - acc: 0.9721 - val_loss: 0.1344 - val_acc: 0.9815            

Epoch 4/30                                                                          
 - 14s - loss: 0.1246 - acc: 0.9755 - val_loss: 0.0732 - val_acc: 0.9894            

Epoch 5/30                                                                          
 - 14s - loss: 0.1027 - acc: 0.9784 - val_loss: 0.2016 - val_acc: 0.9810            

Epoch 6/30                                                                          
 - 14s - loss: 0.1106 - acc: 0.9785 - val_loss: 0.0675 - val_acc: 0.9888            

Epoch 7/30                                                                          
 - 14s - loss: 0.1039 - acc: 0.9795 - val_loss: 0.0651 - val_acc: 0.9910            

Epoch 8/30                                                                          
 - 14s - loss: 0.0937 - acc: 0.9821 - val_loss: 0.0638 - val_acc: 0.9911            

Epoch 9/30                                                                          
 - 14s - loss: 0.0894 - acc: 0.9838 - val_loss: 0.1246 - val_acc: 0.9861            

Epoch 10/30                                                                         
 - 14s - loss: 0.0872 - acc: 0.9834 - val_loss: 0.0754 - val_acc: 0.9921            

Epoch 11/30                                                                         
 - 14s - loss: 0.0593 - acc: 0.9880 - val_loss: 0.0526 - val_acc: 0.9929            

Epoch 12/30                                                                         
 - 14s - loss: 0.0509 - acc: 0.9902 - val_loss: 0.0538 - val_acc: 0.9937            

Epoch 13/30                                                                         
 - 13s - loss: 0.0484 - acc: 0.9904 - val_loss: 0.0543 - val_acc: 0.9945            

Epoch 14/30                                                                         
 - 14s - loss: 0.0418 - acc: 0.9915 - val_loss: 0.0444 - val_acc: 0.9956            

Epoch 15/30                                                                         
 - 14s - loss: 0.0347 - acc: 0.9926 - val_loss: 0.0417 - val_acc: 0.9954            

Epoch 16/30                                                                         
 - 14s - loss: 0.0357 - acc: 0.9925 - val_loss: 0.0501 - val_acc: 0.9949            

Epoch 17/30                                                                         
 - 14s - loss: 0.0323 - acc: 0.9934 - val_loss: 0.0360 - val_acc: 0.9964            

Epoch 18/30                                                                         
 - 14s - loss: 0.0322 - acc: 0.9933 - val_loss: 0.0428 - val_acc: 0.9956            

Epoch 19/30                                                                         
 - 14s - loss: 0.0341 - acc: 0.9940 - val_loss: 0.0427 - val_acc: 0.9951            

Epoch 20/30                                                                         
 - 14s - loss: 0.0329 - acc: 0.9933 - val_loss: 0.0440 - val_acc: 0.9950            

Epoch 1/30                                                                          
 - 23s - loss: 0.7654 - acc: 0.7562 - val_loss: 0.1295 - val_acc: 0.9629            

Epoch 2/30                                                                          
 - 14s - loss: 0.2781 - acc: 0.9148 - val_loss: 0.0799 - val_acc: 0.9763            

Epoch 3/30                                                                          
 - 14s - loss: 0.1998 - acc: 0.9381 - val_loss: 0.0516 - val_acc: 0.9840            

Epoch 4/30                                                                          
 - 14s - loss: 0.1668 - acc: 0.9499 - val_loss: 0.0578 - val_acc: 0.9817            

Epoch 5/30                                                                          
 - 14s - loss: 0.1403 - acc: 0.9565 - val_loss: 0.0448 - val_acc: 0.9863            

Epoch 6/30                                                                          
 - 14s - loss: 0.1217 - acc: 0.9631 - val_loss: 0.0367 - val_acc: 0.9879            

Epoch 7/30                                                                          
 - 14s - loss: 0.1095 - acc: 0.9655 - val_loss: 0.0386 - val_acc: 0.9869            

Epoch 8/30                                                                          
 - 14s - loss: 0.1114 - acc: 0.9677 - val_loss: 0.0338 - val_acc: 0.9890            

Epoch 9/30                                                                          
 - 14s - loss: 0.1009 - acc: 0.9693 - val_loss: 0.0353 - val_acc: 0.9894            

Epoch 10/30                                                                         
 - 14s - loss: 0.0920 - acc: 0.9721 - val_loss: 0.0337 - val_acc: 0.9894            

Epoch 11/30                                                                         
 - 14s - loss: 0.0882 - acc: 0.9727 - val_loss: 0.0312 - val_acc: 0.9914            

Epoch 12/30                                                                         
 - 14s - loss: 0.0813 - acc: 0.9754 - val_loss: 0.0303 - val_acc: 0.9907            

Epoch 13/30                                                                         
 - 14s - loss: 0.0836 - acc: 0.9750 - val_loss: 0.0273 - val_acc: 0.9918            

Epoch 14/30                                                                         
 - 14s - loss: 0.0807 - acc: 0.9765 - val_loss: 0.0306 - val_acc: 0.9914            

Epoch 15/30                                                                         
 - 14s - loss: 0.0722 - acc: 0.9780 - val_loss: 0.0309 - val_acc: 0.9913            

Epoch 16/30                                                                         
 - 14s - loss: 0.0743 - acc: 0.9776 - val_loss: 0.0241 - val_acc: 0.9932            

Epoch 17/30                                                                         
 - 14s - loss: 0.0655 - acc: 0.9806 - val_loss: 0.0246 - val_acc: 0.9924            

Epoch 18/30                                                                         
 - 14s - loss: 0.0690 - acc: 0.9792 - val_loss: 0.0235 - val_acc: 0.9936            

Epoch 19/30                                                                         
 - 14s - loss: 0.0667 - acc: 0.9797 - val_loss: 0.0243 - val_acc: 0.9927            

Epoch 20/30                                                                         
 - 14s - loss: 0.0677 - acc: 0.9796 - val_loss: 0.0275 - val_acc: 0.9924            

Epoch 21/30                                                                         
 - 14s - loss: 0.0658 - acc: 0.9801 - val_loss: 0.0198 - val_acc: 0.9942            

Epoch 22/30                                                                         
 - 14s - loss: 0.0663 - acc: 0.9799 - val_loss: 0.0246 - val_acc: 0.9929            

Epoch 23/30                                                                         
 - 14s - loss: 0.0644 - acc: 0.9807 - val_loss: 0.0273 - val_acc: 0.9923            

Epoch 24/30                                                                         
 - 14s - loss: 0.0650 - acc: 0.9798 - val_loss: 0.0184 - val_acc: 0.9943            

Epoch 25/30                                                                         
 - 14s - loss: 0.0635 - acc: 0.9806 - val_loss: 0.0275 - val_acc: 0.9924            

Epoch 26/30                                                                         
 - 14s - loss: 0.0636 - acc: 0.9811 - val_loss: 0.0245 - val_acc: 0.9927            

Epoch 27/30                                                                         
 - 14s - loss: 0.0599 - acc: 0.9817 - val_loss: 0.0225 - val_acc: 0.9935            

Epoch 1/30                                                                          
 - 24s - loss: 0.3980 - acc: 0.8750 - val_loss: 0.0950 - val_acc: 0.9718            

Epoch 2/30                                                                          
 - 14s - loss: 0.1621 - acc: 0.9497 - val_loss: 0.0514 - val_acc: 0.9833            

Epoch 3/30                                                                          
 - 14s - loss: 0.1247 - acc: 0.9618 - val_loss: 0.0448 - val_acc: 0.9846            

Epoch 4/30                                                                          
 - 14s - loss: 0.1068 - acc: 0.9671 - val_loss: 0.0344 - val_acc: 0.9882            

Epoch 5/30                                                                          
 - 14s - loss: 0.0898 - acc: 0.9729 - val_loss: 0.0422 - val_acc: 0.9873            

Epoch 6/30                                                                          
 - 14s - loss: 0.0751 - acc: 0.9761 - val_loss: 0.0293 - val_acc: 0.9908            

Epoch 7/30                                                                          
 - 14s - loss: 0.0708 - acc: 0.9783 - val_loss: 0.0334 - val_acc: 0.9904            

Epoch 8/30                                                                          
 - 14s - loss: 0.0696 - acc: 0.9784 - val_loss: 0.0306 - val_acc: 0.9900            

Epoch 9/30                                                                          
 - 15s - loss: 0.0629 - acc: 0.9807 - val_loss: 0.0280 - val_acc: 0.9912            

Epoch 10/30                                                                         
 - 14s - loss: 0.0558 - acc: 0.9829 - val_loss: 0.0236 - val_acc: 0.9935            

Epoch 11/30                                                                         
 - 14s - loss: 0.0570 - acc: 0.9826 - val_loss: 0.0267 - val_acc: 0.9915            

Epoch 12/30                                                                         
 - 14s - loss: 0.0530 - acc: 0.9837 - val_loss: 0.0240 - val_acc: 0.9926            

Epoch 13/30                                                                         
 - 14s - loss: 0.0493 - acc: 0.9849 - val_loss: 0.0239 - val_acc: 0.9926            

Epoch 1/30                                                                          
 - 24s - loss: 0.4408 - acc: 0.8600 - val_loss: 0.1750 - val_acc: 0.9555            

Epoch 2/30                                                                          
 - 14s - loss: 0.1783 - acc: 0.9471 - val_loss: 0.1170 - val_acc: 0.9671            

Epoch 3/30                                                                          
 - 14s - loss: 0.1328 - acc: 0.9591 - val_loss: 0.0642 - val_acc: 0.9825            

Epoch 4/30                                                                          
 - 15s - loss: 0.1175 - acc: 0.9656 - val_loss: 0.0693 - val_acc: 0.9798            

Epoch 5/30                                                                          
 - 14s - loss: 0.1004 - acc: 0.9695 - val_loss: 0.0393 - val_acc: 0.9877            

Epoch 6/30                                                                          
 - 14s - loss: 0.0913 - acc: 0.9726 - val_loss: 0.0270 - val_acc: 0.9929            

Epoch 7/30                                                                          
 - 15s - loss: 0.0869 - acc: 0.9737 - val_loss: 0.0312 - val_acc: 0.9901            

Epoch 8/30                                                                          
 - 14s - loss: 0.0786 - acc: 0.9759 - val_loss: 0.0509 - val_acc: 0.9844            

Epoch 9/30                                                                          
 - 14s - loss: 0.0578 - acc: 0.9821 - val_loss: 0.0218 - val_acc: 0.9937            

Epoch 10/30                                                                         
 - 15s - loss: 0.0507 - acc: 0.9845 - val_loss: 0.0238 - val_acc: 0.9940            

Epoch 11/30                                                                         
 - 14s - loss: 0.0442 - acc: 0.9874 - val_loss: 0.0232 - val_acc: 0.9935            

Epoch 12/30                                                                         
 - 14s - loss: 0.0421 - acc: 0.9865 - val_loss: 0.0218 - val_acc: 0.9946            

Epoch 1/30                                                                          
 - 30s - loss: 0.3098 - acc: 0.9004 - val_loss: 0.0635 - val_acc: 0.9801            

Epoch 2/30                                                                          
 - 18s - loss: 0.1144 - acc: 0.9632 - val_loss: 0.0357 - val_acc: 0.9886            

Epoch 3/30                                                                          
 - 18s - loss: 0.0877 - acc: 0.9725 - val_loss: 0.0325 - val_acc: 0.9912            

Epoch 4/30                                                                          
 - 18s - loss: 0.0766 - acc: 0.9758 - val_loss: 0.0341 - val_acc: 0.9894            

Epoch 5/30                                                                          
 - 17s - loss: 0.0676 - acc: 0.9795 - val_loss: 0.0315 - val_acc: 0.9904            

Epoch 6/30                                                                          
 - 18s - loss: 0.0609 - acc: 0.9808 - val_loss: 0.0254 - val_acc: 0.9926            

Epoch 7/30                                                                          
 - 18s - loss: 0.0547 - acc: 0.9824 - val_loss: 0.0238 - val_acc: 0.9927            

Epoch 8/30                                                                          
 - 18s - loss: 0.0498 - acc: 0.9842 - val_loss: 0.0219 - val_acc: 0.9931            

Epoch 9/30                                                                          
 - 18s - loss: 0.0446 - acc: 0.9863 - val_loss: 0.0227 - val_acc: 0.9936            

Epoch 10/30                                                                         
 - 18s - loss: 0.0452 - acc: 0.9856 - val_loss: 0.0219 - val_acc: 0.9930            

Epoch 11/30                                                                         
 - 18s - loss: 0.0389 - acc: 0.9882 - val_loss: 0.0172 - val_acc: 0.9938            

Epoch 12/30                                                                         
 - 18s - loss: 0.0389 - acc: 0.9875 - val_loss: 0.0190 - val_acc: 0.9942            

Epoch 13/30                                                                         
 - 18s - loss: 0.0353 - acc: 0.9887 - val_loss: 0.0137 - val_acc: 0.9950            

Epoch 14/30                                                                         
 - 18s - loss: 0.0371 - acc: 0.9882 - val_loss: 0.0192 - val_acc: 0.9939            

Epoch 15/30                                                                         
 - 18s - loss: 0.0346 - acc: 0.9883 - val_loss: 0.0134 - val_acc: 0.9956            

Epoch 16/30                                                                         
 - 17s - loss: 0.0375 - acc: 0.9878 - val_loss: 0.0182 - val_acc: 0.9946            

Epoch 17/30                                                                         
 - 17s - loss: 0.0325 - acc: 0.9892 - val_loss: 0.0163 - val_acc: 0.9950            

Epoch 18/30                                                                         
 - 17s - loss: 0.0324 - acc: 0.9893 - val_loss: 0.0128 - val_acc: 0.9955            

Epoch 19/30                                                                         
 - 18s - loss: 0.0327 - acc: 0.9901 - val_loss: 0.0175 - val_acc: 0.9948            

Epoch 20/30                                                                         
 - 18s - loss: 0.0318 - acc: 0.9901 - val_loss: 0.0142 - val_acc: 0.9957            

Epoch 21/30                                                                         
 - 17s - loss: 0.0306 - acc: 0.9897 - val_loss: 0.0159 - val_acc: 0.9954            

Epoch 1/30                                                                          
 - 27s - loss: 0.5833 - acc: 0.9192 - val_loss: 0.2125 - val_acc: 0.9798            

Epoch 2/30                                                                          
 - 15s - loss: 0.3753 - acc: 0.9613 - val_loss: 0.2395 - val_acc: 0.9799            

Epoch 3/30                                                                          
 - 15s - loss: 0.3165 - acc: 0.9688 - val_loss: 0.1753 - val_acc: 0.9867            

Epoch 4/30                                                                          
 - 16s - loss: 0.2935 - acc: 0.9721 - val_loss: 0.1860 - val_acc: 0.9831            

Epoch 5/30                                                                          
 - 15s - loss: 0.3030 - acc: 0.9725 - val_loss: 0.3114 - val_acc: 0.9771            

Epoch 6/30                                                                          
 - 15s - loss: 0.1678 - acc: 0.9840 - val_loss: 0.0845 - val_acc: 0.9924            

Epoch 7/30                                                                          
 - 15s - loss: 0.1404 - acc: 0.9866 - val_loss: 0.0913 - val_acc: 0.9921            

Epoch 8/30                                                                          
 - 16s - loss: 0.1245 - acc: 0.9863 - val_loss: 0.0976 - val_acc: 0.9907            

Epoch 9/30                                                                          
 - 16s - loss: 0.0855 - acc: 0.9893 - val_loss: 0.0605 - val_acc: 0.9929            

Epoch 10/30                                                                         
 - 15s - loss: 0.0825 - acc: 0.9898 - val_loss: 0.0573 - val_acc: 0.9932            

Epoch 11/30                                                                         
 - 15s - loss: 0.0729 - acc: 0.9907 - val_loss: 0.0485 - val_acc: 0.9939            

Epoch 12/30                                                                         
 - 15s - loss: 0.0691 - acc: 0.9906 - val_loss: 0.0452 - val_acc: 0.9943            

Epoch 13/30                                                                         
 - 15s - loss: 0.0644 - acc: 0.9914 - val_loss: 0.0360 - val_acc: 0.9957            

Epoch 14/30                                                                         
 - 15s - loss: 0.0658 - acc: 0.9912 - val_loss: 0.0559 - val_acc: 0.9939            

Epoch 15/30                                                                         
 - 15s - loss: 0.0572 - acc: 0.9924 - val_loss: 0.0390 - val_acc: 0.9948            

Epoch 16/30                                                                         
 - 15s - loss: 0.0488 - acc: 0.9924 - val_loss: 0.0381 - val_acc: 0.9951            

Epoch 1/30                                                                          
 - 26s - loss: 0.2547 - acc: 0.9170 - val_loss: 0.0509 - val_acc: 0.9840            

Epoch 2/30                                                                          
 - 15s - loss: 0.1018 - acc: 0.9680 - val_loss: 0.0492 - val_acc: 0.9854            

Epoch 3/30                                                                          
 - 15s - loss: 0.0717 - acc: 0.9766 - val_loss: 0.0340 - val_acc: 0.9894            

Epoch 4/30                                                                          
 - 15s - loss: 0.0644 - acc: 0.9796 - val_loss: 0.0295 - val_acc: 0.9912            

Epoch 5/30                                                                          
 - 15s - loss: 0.0508 - acc: 0.9833 - val_loss: 0.0249 - val_acc: 0.9921            

Epoch 6/30                                                                          
 - 14s - loss: 0.0523 - acc: 0.9834 - val_loss: 0.0188 - val_acc: 0.9937            

Epoch 7/30                                                                          
 - 14s - loss: 0.0470 - acc: 0.9853 - val_loss: 0.0235 - val_acc: 0.9929            

Epoch 8/30                                                                          
 - 14s - loss: 0.0413 - acc: 0.9866 - val_loss: 0.0316 - val_acc: 0.9914            

Epoch 9/30                                                                          
 - 15s - loss: 0.0368 - acc: 0.9882 - val_loss: 0.0146 - val_acc: 0.9955            

Epoch 10/30                                                                         
 - 14s - loss: 0.0344 - acc: 0.9896 - val_loss: 0.0225 - val_acc: 0.9932            

Epoch 11/30                                                                         
 - 15s - loss: 0.0320 - acc: 0.9898 - val_loss: 0.0226 - val_acc: 0.9939            

Epoch 12/30                                                                         
 - 14s - loss: 0.0316 - acc: 0.9902 - val_loss: 0.0199 - val_acc: 0.9939            

Epoch 1/30                                                                          
 - 28s - loss: 0.1723 - acc: 0.9469 - val_loss: 0.0918 - val_acc: 0.9721            

Epoch 2/30                                                                          
 - 14s - loss: 0.0690 - acc: 0.9771 - val_loss: 0.0359 - val_acc: 0.9886            

Epoch 3/30                                                                          
 - 14s - loss: 0.0530 - acc: 0.9827 - val_loss: 0.0342 - val_acc: 0.9892            

Epoch 4/30                                                                          
 - 14s - loss: 0.0442 - acc: 0.9860 - val_loss: 0.0211 - val_acc: 0.9932            

Epoch 5/30                                                                          
 - 15s - loss: 0.0411 - acc: 0.9873 - val_loss: 0.0277 - val_acc: 0.9902            

Epoch 6/30                                                                          
 - 14s - loss: 0.0342 - acc: 0.9897 - val_loss: 0.0252 - val_acc: 0.9911            

Epoch 7/30                                                                          
 - 15s - loss: 0.0279 - acc: 0.9907 - val_loss: 0.0175 - val_acc: 0.9943            

Epoch 8/30                                                                          
 - 15s - loss: 0.0236 - acc: 0.9923 - val_loss: 0.0196 - val_acc: 0.9940            

Epoch 9/30                                                                          
 - 14s - loss: 0.0232 - acc: 0.9925 - val_loss: 0.0207 - val_acc: 0.9940            

Epoch 10/30                                                                         
 - 14s - loss: 0.0220 - acc: 0.9932 - val_loss: 0.0205 - val_acc: 0.9935            

Epoch 1/30                                                                          
 - 27s - loss: 0.2539 - acc: 0.9189 - val_loss: 0.0575 - val_acc: 0.9827            

Epoch 2/30                                                                          
 - 15s - loss: 0.1015 - acc: 0.9676 - val_loss: 0.0413 - val_acc: 0.9871            

Epoch 3/30                                                                          
 - 14s - loss: 0.0792 - acc: 0.9749 - val_loss: 0.0306 - val_acc: 0.9914            

Epoch 4/30                                                                          
 - 14s - loss: 0.0676 - acc: 0.9783 - val_loss: 0.0350 - val_acc: 0.9905            

Epoch 5/30                                                                          
 - 15s - loss: 0.0535 - acc: 0.9830 - val_loss: 0.0236 - val_acc: 0.9929            

Epoch 6/30                                                                          
 - 14s - loss: 0.0477 - acc: 0.9847 - val_loss: 0.0280 - val_acc: 0.9918            

Epoch 7/30                                                                          
 - 14s - loss: 0.0470 - acc: 0.9852 - val_loss: 0.0212 - val_acc: 0.9935            

Epoch 8/30                                                                          
 - 14s - loss: 0.0419 - acc: 0.9869 - val_loss: 0.0222 - val_acc: 0.9943            

Epoch 9/30                                                                          
 - 14s - loss: 0.0390 - acc: 0.9870 - val_loss: 0.0247 - val_acc: 0.9921            

Epoch 10/30                                                                         
 - 15s - loss: 0.0360 - acc: 0.9885 - val_loss: 0.0207 - val_acc: 0.9943            

Epoch 11/30                                                                         
 - 14s - loss: 0.0327 - acc: 0.9897 - val_loss: 0.0197 - val_acc: 0.9935            

Epoch 12/30                                                                         
 - 14s - loss: 0.0318 - acc: 0.9901 - val_loss: 0.0190 - val_acc: 0.9940            

Epoch 13/30                                                                         
 - 14s - loss: 0.0299 - acc: 0.9901 - val_loss: 0.0212 - val_acc: 0.9942            

Epoch 14/30                                                                         
 - 14s - loss: 0.0310 - acc: 0.9910 - val_loss: 0.0180 - val_acc: 0.9943            

Epoch 15/30                                                                         
 - 14s - loss: 0.0274 - acc: 0.9909 - val_loss: 0.0190 - val_acc: 0.9943            

Epoch 16/30                                                                         
 - 14s - loss: 0.0297 - acc: 0.9902 - val_loss: 0.0173 - val_acc: 0.9940            

Epoch 17/30                                                                         
 - 15s - loss: 0.0283 - acc: 0.9905 - val_loss: 0.0190 - val_acc: 0.9940            

Epoch 18/30                                                                         
 - 15s - loss: 0.0288 - acc: 0.9910 - val_loss: 0.0166 - val_acc: 0.9949            

Epoch 19/30                                                                         
 - 15s - loss: 0.0275 - acc: 0.9908 - val_loss: 0.0168 - val_acc: 0.9950            

Epoch 20/30                                                                         
 - 15s - loss: 0.0269 - acc: 0.9919 - val_loss: 0.0158 - val_acc: 0.9954            

Epoch 21/30                                                                         
 - 15s - loss: 0.0259 - acc: 0.9917 - val_loss: 0.0181 - val_acc: 0.9943            

Epoch 22/30                                                                         
 - 15s - loss: 0.0275 - acc: 0.9913 - val_loss: 0.0151 - val_acc: 0.9956            

Epoch 23/30                                                                         
 - 15s - loss: 0.0254 - acc: 0.9918 - val_loss: 0.0212 - val_acc: 0.9931            

Epoch 24/30                                                                         
 - 15s - loss: 0.0257 - acc: 0.9922 - val_loss: 0.0167 - val_acc: 0.9951            

Epoch 25/30                                                                         
 - 15s - loss: 0.0243 - acc: 0.9915 - val_loss: 0.0186 - val_acc: 0.9944            

Epoch 1/30                                                                          
 - 27s - loss: 0.2612 - acc: 0.9184 - val_loss: 0.0539 - val_acc: 0.9833            

Epoch 2/30                                                                          
 - 14s - loss: 0.1021 - acc: 0.9677 - val_loss: 0.0409 - val_acc: 0.9879            

Epoch 3/30                                                                          
 - 14s - loss: 0.0720 - acc: 0.9781 - val_loss: 0.0278 - val_acc: 0.9902            

Epoch 4/30                                                                          
 - 14s - loss: 0.0647 - acc: 0.9785 - val_loss: 0.0295 - val_acc: 0.9920            

Epoch 5/30                                                                          
 - 15s - loss: 0.0559 - acc: 0.9820 - val_loss: 0.0308 - val_acc: 0.9904            

Epoch 6/30                                                                          
 - 15s - loss: 0.0500 - acc: 0.9844 - val_loss: 0.0211 - val_acc: 0.9937            

Epoch 7/30                                                                          
 - 15s - loss: 0.0453 - acc: 0.9854 - val_loss: 0.0219 - val_acc: 0.9933            

Epoch 8/30                                                                          
 - 15s - loss: 0.0433 - acc: 0.9857 - val_loss: 0.0225 - val_acc: 0.9935            

Epoch 9/30                                                                          
 - 15s - loss: 0.0417 - acc: 0.9864 - val_loss: 0.0217 - val_acc: 0.9931            

Epoch 1/30                                                                          
 - 31s - loss: 0.2414 - acc: 0.9236 - val_loss: 0.0545 - val_acc: 0.9825            

Epoch 2/30                                                                          
 - 17s - loss: 0.0993 - acc: 0.9687 - val_loss: 0.0492 - val_acc: 0.9857            

Epoch 3/30                                                                          
 - 17s - loss: 0.0761 - acc: 0.9757 - val_loss: 0.0295 - val_acc: 0.9910            

Epoch 4/30                                                                          
 - 17s - loss: 0.0605 - acc: 0.9812 - val_loss: 0.0292 - val_acc: 0.9917            

Epoch 5/30                                                                          
 - 17s - loss: 0.0532 - acc: 0.9825 - val_loss: 0.0253 - val_acc: 0.9921            

Epoch 6/30                                                                          
 - 17s - loss: 0.0452 - acc: 0.9859 - val_loss: 0.0246 - val_acc: 0.9919            

Epoch 7/30                                                                          
 - 17s - loss: 0.0452 - acc: 0.9854 - val_loss: 0.0266 - val_acc: 0.9923            

Epoch 8/30                                                                          
 - 18s - loss: 0.0399 - acc: 0.9871 - val_loss: 0.0199 - val_acc: 0.9936            

Epoch 9/30                                                                          
 - 17s - loss: 0.0381 - acc: 0.9880 - val_loss: 0.0228 - val_acc: 0.9933            

Epoch 10/30                                                                         
 - 17s - loss: 0.0362 - acc: 0.9885 - val_loss: 0.0231 - val_acc: 0.9926            

Epoch 11/30                                                                         
 - 17s - loss: 0.0343 - acc: 0.9894 - val_loss: 0.0175 - val_acc: 0.9942            

Epoch 12/30                                                                         
 - 17s - loss: 0.0312 - acc: 0.9906 - val_loss: 0.0188 - val_acc: 0.9946            

Epoch 13/30                                                                         
 - 17s - loss: 0.0314 - acc: 0.9900 - val_loss: 0.0204 - val_acc: 0.9944            

Epoch 14/30                                                                         
 - 17s - loss: 0.0269 - acc: 0.9913 - val_loss: 0.0196 - val_acc: 0.9938            

Epoch 1/30                                                                          
 - 28s - loss: 0.1715 - acc: 0.9469 - val_loss: 0.0524 - val_acc: 0.9857            

Epoch 2/30                                                                          
 - 15s - loss: 0.0670 - acc: 0.9789 - val_loss: 0.0355 - val_acc: 0.9886            

Epoch 3/30                                                                          
 - 15s - loss: 0.0565 - acc: 0.9823 - val_loss: 0.0417 - val_acc: 0.9876            

Epoch 4/30                                                                          
 - 15s - loss: 0.0458 - acc: 0.9862 - val_loss: 0.0297 - val_acc: 0.9912            

Epoch 5/30                                                                          
 - 15s - loss: 0.0362 - acc: 0.9889 - val_loss: 0.0251 - val_acc: 0.9924            

Epoch 6/30                                                                          
 - 15s - loss: 0.0315 - acc: 0.9902 - val_loss: 0.0254 - val_acc: 0.9924            

Epoch 7/30                                                                          
 - 15s - loss: 0.0318 - acc: 0.9903 - val_loss: 0.0232 - val_acc: 0.9926            

Epoch 8/30                                                                          
 - 15s - loss: 0.0280 - acc: 0.9905 - val_loss: 0.0223 - val_acc: 0.9933            

Epoch 9/30                                                                          
 - 14s - loss: 0.0270 - acc: 0.9916 - val_loss: 0.0210 - val_acc: 0.9933            

Epoch 10/30                                                                         
 - 15s - loss: 0.0235 - acc: 0.9927 - val_loss: 0.0245 - val_acc: 0.9927            

Epoch 11/30                                                                         
 - 15s - loss: 0.0224 - acc: 0.9931 - val_loss: 0.0188 - val_acc: 0.9956            

Epoch 12/30                                                                         
 - 15s - loss: 0.0211 - acc: 0.9932 - val_loss: 0.0216 - val_acc: 0.9931            

Epoch 13/30                                                                         
 - 15s - loss: 0.0202 - acc: 0.9935 - val_loss: 0.0229 - val_acc: 0.9935            

Epoch 14/30                                                                         
 - 15s - loss: 0.0177 - acc: 0.9944 - val_loss: 0.0146 - val_acc: 0.9957            

Epoch 15/30                                                                         
 - 15s - loss: 0.0151 - acc: 0.9953 - val_loss: 0.0235 - val_acc: 0.9939            

Epoch 16/30                                                                         
 - 15s - loss: 0.0149 - acc: 0.9954 - val_loss: 0.0141 - val_acc: 0.9955            

Epoch 17/30                                                                         
 - 15s - loss: 0.0165 - acc: 0.9951 - val_loss: 0.0184 - val_acc: 0.9945            

Epoch 18/30                                                                         
 - 15s - loss: 0.0147 - acc: 0.9952 - val_loss: 0.0176 - val_acc: 0.9952            

Epoch 19/30                                                                         
 - 15s - loss: 0.0151 - acc: 0.9956 - val_loss: 0.0175 - val_acc: 0.9951            

Epoch 1/30                                                                          
 - 29s - loss: 0.2600 - acc: 0.9187 - val_loss: 0.0584 - val_acc: 0.9812            

Epoch 2/30                                                                          
 - 15s - loss: 0.1012 - acc: 0.9680 - val_loss: 0.0529 - val_acc: 0.9851            

Epoch 3/30                                                                          
 - 15s - loss: 0.0765 - acc: 0.9761 - val_loss: 0.0409 - val_acc: 0.9901            

Epoch 4/30                                                                          
 - 15s - loss: 0.0656 - acc: 0.9799 - val_loss: 0.0251 - val_acc: 0.9921            

Epoch 5/30                                                                          
 - 15s - loss: 0.0569 - acc: 0.9821 - val_loss: 0.0341 - val_acc: 0.9904            

Epoch 6/30                                                                          
 - 15s - loss: 0.0520 - acc: 0.9841 - val_loss: 0.0209 - val_acc: 0.9937            

Epoch 7/30                                                                          
 - 16s - loss: 0.0456 - acc: 0.9854 - val_loss: 0.0276 - val_acc: 0.9918            

Epoch 8/30                                                                          
 - 15s - loss: 0.0459 - acc: 0.9855 - val_loss: 0.0247 - val_acc: 0.9925            

Epoch 9/30                                                                          
 - 15s - loss: 0.0383 - acc: 0.9876 - val_loss: 0.0219 - val_acc: 0.9930            

Epoch 1/30                                                                          
 - 29s - loss: 0.5410 - acc: 0.8257 - val_loss: 0.0846 - val_acc: 0.9732            

Epoch 2/30                                                                          
 - 15s - loss: 0.1835 - acc: 0.9438 - val_loss: 0.0549 - val_acc: 0.9832            

Epoch 3/30                                                                          
 - 15s - loss: 0.1286 - acc: 0.9605 - val_loss: 0.0496 - val_acc: 0.9850            

Epoch 4/30                                                                          
 - 15s - loss: 0.1089 - acc: 0.9670 - val_loss: 0.0376 - val_acc: 0.9880            

Epoch 5/30                                                                          
 - 15s - loss: 0.0955 - acc: 0.9707 - val_loss: 0.0380 - val_acc: 0.9874            

Epoch 6/30                                                                          
 - 15s - loss: 0.0814 - acc: 0.9741 - val_loss: 0.0307 - val_acc: 0.9893            

Epoch 7/30                                                                          
 - 16s - loss: 0.0809 - acc: 0.9749 - val_loss: 0.0490 - val_acc: 0.9855            

Epoch 8/30                                                                          
 - 15s - loss: 0.0723 - acc: 0.9768 - val_loss: 0.0297 - val_acc: 0.9896            

Epoch 9/30                                                                          
 - 15s - loss: 0.0669 - acc: 0.9783 - val_loss: 0.0276 - val_acc: 0.9918            

Epoch 10/30                                                                         
 - 15s - loss: 0.0611 - acc: 0.9806 - val_loss: 0.0310 - val_acc: 0.9889            

Epoch 11/30                                                                         
 - 15s - loss: 0.0603 - acc: 0.9812 - val_loss: 0.0387 - val_acc: 0.9900            

Epoch 12/30                                                                         
 - 15s - loss: 0.0513 - acc: 0.9838 - val_loss: 0.0245 - val_acc: 0.9924            

Epoch 13/30                                                                         
 - 15s - loss: 0.0521 - acc: 0.9844 - val_loss: 0.0201 - val_acc: 0.9935            

Epoch 14/30                                                                         
 - 15s - loss: 0.0533 - acc: 0.9838 - val_loss: 0.0261 - val_acc: 0.9925            

Epoch 15/30                                                                         
 - 15s - loss: 0.0453 - acc: 0.9858 - val_loss: 0.0223 - val_acc: 0.9927            

Epoch 16/30                                                                         
 - 15s - loss: 0.0484 - acc: 0.9851 - val_loss: 0.0218 - val_acc: 0.9932            

Epoch 1/30                                                                          
 - 29s - loss: 0.4198 - acc: 0.8675 - val_loss: 0.0541 - val_acc: 0.9825            

Epoch 2/30                                                                          
 - 15s - loss: 0.1570 - acc: 0.9512 - val_loss: 0.0437 - val_acc: 0.9858            

Epoch 3/30                                                                          
 - 15s - loss: 0.1191 - acc: 0.9624 - val_loss: 0.0337 - val_acc: 0.9890            

Epoch 4/30                                                                          
 - 15s - loss: 0.0992 - acc: 0.9684 - val_loss: 0.0347 - val_acc: 0.9889            

Epoch 5/30                                                                          
 - 15s - loss: 0.0865 - acc: 0.9735 - val_loss: 0.0412 - val_acc: 0.9879            

Epoch 6/30                                                                          
 - 15s - loss: 0.0780 - acc: 0.9763 - val_loss: 0.0242 - val_acc: 0.9923            

Epoch 7/30                                                                          
 - 15s - loss: 0.0688 - acc: 0.9781 - val_loss: 0.0266 - val_acc: 0.9924            

Epoch 8/30                                                                          
 - 15s - loss: 0.0733 - acc: 0.9769 - val_loss: 0.0247 - val_acc: 0.9925            

Epoch 9/30                                                                          
 - 15s - loss: 0.0648 - acc: 0.9794 - val_loss: 0.0235 - val_acc: 0.9929            

Epoch 10/30                                                                         
 - 16s - loss: 0.0675 - acc: 0.9789 - val_loss: 0.0257 - val_acc: 0.9924            

Epoch 11/30                                                                         
 - 15s - loss: 0.0642 - acc: 0.9795 - val_loss: 0.0243 - val_acc: 0.9930            

Epoch 12/30                                                                         
 - 15s - loss: 0.0666 - acc: 0.9792 - val_loss: 0.0257 - val_acc: 0.9924            

Epoch 1/30                                                                          
 - 29s - loss: 0.2276 - acc: 0.9388 - val_loss: 0.0552 - val_acc: 0.9852            

Epoch 2/30                                                                          
 - 15s - loss: 0.0762 - acc: 0.9790 - val_loss: 0.0507 - val_acc: 0.9836            

Epoch 3/30                                                                          
 - 15s - loss: 0.0549 - acc: 0.9841 - val_loss: 0.0339 - val_acc: 0.9898            

Epoch 4/30                                                                          
 - 15s - loss: 0.0483 - acc: 0.9865 - val_loss: 0.0323 - val_acc: 0.9894            

Epoch 5/30                                                                          
 - 16s - loss: 0.0413 - acc: 0.9882 - val_loss: 0.0358 - val_acc: 0.9896            

Epoch 6/30                                                                          
 - 15s - loss: 0.0362 - acc: 0.9888 - val_loss: 0.0319 - val_acc: 0.9902            

Epoch 7/30                                                                          
 - 15s - loss: 0.0340 - acc: 0.9898 - val_loss: 0.0289 - val_acc: 0.9913            

Epoch 8/30                                                                          
 - 15s - loss: 0.0314 - acc: 0.9907 - val_loss: 0.0252 - val_acc: 0.9918            

Epoch 9/30                                                                          
 - 15s - loss: 0.0283 - acc: 0.9919 - val_loss: 0.0235 - val_acc: 0.9927            

Epoch 10/30                                                                         
 - 15s - loss: 0.0285 - acc: 0.9908 - val_loss: 0.0225 - val_acc: 0.9930            

Epoch 11/30                                                                         
 - 15s - loss: 0.0264 - acc: 0.9923 - val_loss: 0.0183 - val_acc: 0.9948            

Epoch 12/30                                                                         
 - 15s - loss: 0.0235 - acc: 0.9927 - val_loss: 0.0218 - val_acc: 0.9932            

Epoch 13/30                                                                         
 - 15s - loss: 0.0221 - acc: 0.9932 - val_loss: 0.0203 - val_acc: 0.9930            

Epoch 14/30                                                                         
 - 16s - loss: 0.0188 - acc: 0.9947 - val_loss: 0.0170 - val_acc: 0.9944            

Epoch 15/30                                                                         
 - 15s - loss: 0.0162 - acc: 0.9954 - val_loss: 0.0175 - val_acc: 0.9946            

Epoch 16/30                                                                         
 - 15s - loss: 0.0165 - acc: 0.9953 - val_loss: 0.0174 - val_acc: 0.9946            

Epoch 17/30                                                                         
 - 15s - loss: 0.0150 - acc: 0.9956 - val_loss: 0.0166 - val_acc: 0.9951            

Epoch 18/30                                                                         
 - 15s - loss: 0.0139 - acc: 0.9963 - val_loss: 0.0159 - val_acc: 0.9951            

Epoch 19/30                                                                         
 - 16s - loss: 0.0150 - acc: 0.9957 - val_loss: 0.0169 - val_acc: 0.9949            

Epoch 20/30                                                                         
 - 15s - loss: 0.0153 - acc: 0.9959 - val_loss: 0.0166 - val_acc: 0.9951            

Epoch 21/30                                                                         
 - 15s - loss: 0.0148 - acc: 0.9956 - val_loss: 0.0165 - val_acc: 0.9950            

Epoch 1/30                                                                          
 - 30s - loss: 0.4987 - acc: 0.8427 - val_loss: 0.1087 - val_acc: 0.9671            

Epoch 2/30                                                                          
 - 15s - loss: 0.1680 - acc: 0.9492 - val_loss: 0.0691 - val_acc: 0.9795            

Epoch 3/30                                                                          
 - 15s - loss: 0.1223 - acc: 0.9624 - val_loss: 0.0353 - val_acc: 0.9888            

Epoch 4/30                                                                          
 - 15s - loss: 0.1006 - acc: 0.9689 - val_loss: 0.0487 - val_acc: 0.9848            

Epoch 5/30                                                                          
 - 15s - loss: 0.0887 - acc: 0.9727 - val_loss: 0.0361 - val_acc: 0.9892            

Epoch 6/30                                                                          
 - 15s - loss: 0.0773 - acc: 0.9755 - val_loss: 0.0344 - val_acc: 0.9898            

Epoch 7/30                                                                          
 - 15s - loss: 0.0712 - acc: 0.9777 - val_loss: 0.0250 - val_acc: 0.9921            

Epoch 8/30                                                                          
 - 15s - loss: 0.0721 - acc: 0.9776 - val_loss: 0.0332 - val_acc: 0.9898            

Epoch 9/30                                                                          
 - 15s - loss: 0.0685 - acc: 0.9786 - val_loss: 0.0296 - val_acc: 0.9905            

Epoch 10/30                                                                         
 - 15s - loss: 0.0676 - acc: 0.9786 - val_loss: 0.0286 - val_acc: 0.9906            

Epoch 1/30                                                                          
 - 35s - loss: 0.3322 - acc: 0.9121 - val_loss: 0.1658 - val_acc: 0.9780            

Epoch 2/30                                                                          
 - 18s - loss: 0.1908 - acc: 0.9568 - val_loss: 0.1393 - val_acc: 0.9829            

Epoch 3/30                                                                          
 - 18s - loss: 0.1404 - acc: 0.9679 - val_loss: 0.1026 - val_acc: 0.9850            

Epoch 4/30                                                                          
 - 19s - loss: 0.1349 - acc: 0.9701 - val_loss: 0.1686 - val_acc: 0.9743            

Epoch 5/30                                                                          
 - 18s - loss: 0.1292 - acc: 0.9723 - val_loss: 0.0548 - val_acc: 0.9899            

Epoch 6/30                                                                          
 - 18s - loss: 0.1349 - acc: 0.9721 - val_loss: 0.0624 - val_acc: 0.9906            

Epoch 7/30                                                                          
 - 19s - loss: 0.1141 - acc: 0.9757 - val_loss: 0.0719 - val_acc: 0.9886            

Epoch 8/30                                                                          
 - 19s - loss: 0.0845 - acc: 0.9818 - val_loss: 0.0561 - val_acc: 0.9912            

Epoch 1/30                                                                          
 - 31s - loss: 0.4166 - acc: 0.8683 - val_loss: 0.0654 - val_acc: 0.9773            

Epoch 2/30                                                                          
 - 16s - loss: 0.1473 - acc: 0.9548 - val_loss: 0.0670 - val_acc: 0.9776            

Epoch 3/30                                                                          
 - 15s - loss: 0.1055 - acc: 0.9675 - val_loss: 0.0515 - val_acc: 0.9836            

Epoch 4/30                                                                          
 - 16s - loss: 0.0903 - acc: 0.9727 - val_loss: 0.0366 - val_acc: 0.9875            

Epoch 5/30                                                                          
 - 15s - loss: 0.0756 - acc: 0.9767 - val_loss: 0.0440 - val_acc: 0.9864            

Epoch 6/30                                                                          
 - 15s - loss: 0.0701 - acc: 0.9784 - val_loss: 0.0389 - val_acc: 0.9871            

Epoch 7/30                                                                          
 - 16s - loss: 0.0595 - acc: 0.9813 - val_loss: 0.0276 - val_acc: 0.9914            

Epoch 8/30                                                                          
 - 15s - loss: 0.0591 - acc: 0.9818 - val_loss: 0.0249 - val_acc: 0.9918            

Epoch 9/30                                                                          
 - 16s - loss: 0.0532 - acc: 0.9833 - val_loss: 0.0252 - val_acc: 0.9904            

Epoch 10/30                                                                         
 - 15s - loss: 0.0537 - acc: 0.9829 - val_loss: 0.0304 - val_acc: 0.9910            

Epoch 11/30                                                                         
 - 15s - loss: 0.0500 - acc: 0.9841 - val_loss: 0.0249 - val_acc: 0.9912            

Epoch 1/30                                                                          
 - 32s - loss: 0.4039 - acc: 0.8804 - val_loss: 0.0770 - val_acc: 0.9777          

Epoch 2/30                                                                        
 - 15s - loss: 0.1363 - acc: 0.9582 - val_loss: 0.0777 - val_acc: 0.9730          

Epoch 3/30                                                                        
 - 15s - loss: 0.0984 - acc: 0.9692 - val_loss: 0.0796 - val_acc: 0.9790          

Epoch 4/30                                                                        
 - 15s - loss: 0.0857 - acc: 0.9746 - val_loss: 0.0329 - val_acc: 0.9898          

Epoch 5/30                                                                        
 - 16s - loss: 0.0776 - acc: 0.9769 - val_loss: 0.0379 - val_acc: 0.9886          

Epoch 6/30                                                                        
 - 15s - loss: 0.0727 - acc: 0.9785 - val_loss: 0.0391 - val_acc: 0.9885          

Epoch 7/30                                                                        
 - 16s - loss: 0.0719 - acc: 0.9790 - val_loss: 0.0335 - val_acc: 0.9895          

Epoch 1/30                                                                        
 - 32s - loss: 0.3816 - acc: 0.9165 - val_loss: 0.3787 - val_acc: 0.9611          

Epoch 2/30                                                                        
 - 16s - loss: 0.1981 - acc: 0.9608 - val_loss: 0.1979 - val_acc: 0.9800          

Epoch 3/30                                                                        
 - 16s - loss: 0.1507 - acc: 0.9689 - val_loss: 0.1114 - val_acc: 0.9861          

Epoch 4/30                                                                        
 - 15s - loss: 0.1342 - acc: 0.9738 - val_loss: 0.1051 - val_acc: 0.9807          

Epoch 5/30                                                                        
 - 16s - loss: 0.1183 - acc: 0.9765 - val_loss: 0.0787 - val_acc: 0.9856          

Epoch 6/30                                                                        
 - 16s - loss: 0.1176 - acc: 0.9783 - val_loss: 0.1074 - val_acc: 0.9871          

Epoch 7/30                                                                        
 - 16s - loss: 0.1066 - acc: 0.9790 - val_loss: 0.0739 - val_acc: 0.9918          

Epoch 8/30                                                                        
 - 16s - loss: 0.0992 - acc: 0.9804 - val_loss: 0.0884 - val_acc: 0.9904          

Epoch 9/30                                                                        
 - 15s - loss: 0.1008 - acc: 0.9819 - val_loss: 0.0958 - val_acc: 0.9868          

Epoch 10/30                                                                       
 - 16s - loss: 0.0691 - acc: 0.9865 - val_loss: 0.0395 - val_acc: 0.9942          

Epoch 11/30                                                                       
 - 16s - loss: 0.0565 - acc: 0.9891 - val_loss: 0.0358 - val_acc: 0.9951          

Epoch 12/30                                                                       
 - 16s - loss: 0.0511 - acc: 0.9895 - val_loss: 0.0427 - val_acc: 0.9945          

Epoch 13/30                                                                       
 - 16s - loss: 0.0561 - acc: 0.9897 - val_loss: 0.0484 - val_acc: 0.9937          

Epoch 14/30                                                                       
 - 16s - loss: 0.0473 - acc: 0.9910 - val_loss: 0.0385 - val_acc: 0.9950          

Epoch 1/30                                                                        
 - 43s - loss: 0.2885 - acc: 0.9180 - val_loss: 0.0806 - val_acc: 0.9770          

Epoch 2/30                                                                        
 - 23s - loss: 0.1190 - acc: 0.9667 - val_loss: 0.0585 - val_acc: 0.9831          

Epoch 3/30                                                                        
 - 24s - loss: 0.0853 - acc: 0.9759 - val_loss: 0.0487 - val_acc: 0.9881          

Epoch 4/30                                                                        
 - 23s - loss: 0.0723 - acc: 0.9795 - val_loss: 0.0491 - val_acc: 0.9868          

Epoch 5/30                                                                        
 - 23s - loss: 0.0649 - acc: 0.9807 - val_loss: 0.0384 - val_acc: 0.9907          

Epoch 6/30                                                                        
 - 23s - loss: 0.0579 - acc: 0.9832 - val_loss: 0.0334 - val_acc: 0.9918          

Epoch 7/30                                                                        
 - 23s - loss: 0.0536 - acc: 0.9842 - val_loss: 0.0423 - val_acc: 0.9895          

Epoch 8/30                                                                        
 - 23s - loss: 0.0445 - acc: 0.9871 - val_loss: 0.0300 - val_acc: 0.9935          

Epoch 9/30                                                                        
 - 23s - loss: 0.0443 - acc: 0.9869 - val_loss: 0.0312 - val_acc: 0.9929          

Epoch 10/30                                                                       
 - 23s - loss: 0.0402 - acc: 0.9876 - val_loss: 0.0301 - val_acc: 0.9924          

Epoch 11/30                                                                       
 - 24s - loss: 0.0341 - acc: 0.9895 - val_loss: 0.0222 - val_acc: 0.9933          

Epoch 12/30                                                                       
 - 23s - loss: 0.0304 - acc: 0.9904 - val_loss: 0.0240 - val_acc: 0.9932          

Epoch 13/30                                                                       
 - 23s - loss: 0.0311 - acc: 0.9911 - val_loss: 0.0250 - val_acc: 0.9935          

Epoch 14/30                                                                       
 - 23s - loss: 0.0262 - acc: 0.9920 - val_loss: 0.0222 - val_acc: 0.9935          

Epoch 15/30                                                                       
 - 23s - loss: 0.0277 - acc: 0.9917 - val_loss: 0.0213 - val_acc: 0.9938          

Epoch 16/30                                                                       
 - 24s - loss: 0.0247 - acc: 0.9926 - val_loss: 0.0214 - val_acc: 0.9939          

Epoch 17/30                                                                       
 - 24s - loss: 0.0248 - acc: 0.9926 - val_loss: 0.0217 - val_acc: 0.9940          

Epoch 18/30                                                                       
 - 24s - loss: 0.0253 - acc: 0.9921 - val_loss: 0.0210 - val_acc: 0.9944          

Epoch 19/30                                                                       
 - 24s - loss: 0.0239 - acc: 0.9929 - val_loss: 0.0205 - val_acc: 0.9943          

Epoch 20/30                                                                       
 - 23s - loss: 0.0240 - acc: 0.9923 - val_loss: 0.0204 - val_acc: 0.9943          

Epoch 21/30                                                                       
 - 24s - loss: 0.0243 - acc: 0.9920 - val_loss: 0.0204 - val_acc: 0.9942          

Epoch 22/30                                                                       
 - 24s - loss: 0.0236 - acc: 0.9924 - val_loss: 0.0205 - val_acc: 0.9944          

Epoch 23/30                                                                       
 - 23s - loss: 0.0269 - acc: 0.9926 - val_loss: 0.0204 - val_acc: 0.9944          

Epoch 24/30                                                                       
 - 24s - loss: 0.0265 - acc: 0.9915 - val_loss: 0.0205 - val_acc: 0.9946          

Epoch 25/30                                                                       
 - 23s - loss: 0.0253 - acc: 0.9925 - val_loss: 0.0205 - val_acc: 0.9944          

Epoch 26/30                                                                       
 - 24s - loss: 0.0248 - acc: 0.9925 - val_loss: 0.0205 - val_acc: 0.9944          

Epoch 1/30                                                                          
 - 33s - loss: 0.8574 - acc: 0.7248 - val_loss: 0.1646 - val_acc: 0.9519            

Epoch 2/30                                                                          
 - 16s - loss: 0.3019 - acc: 0.9083 - val_loss: 0.1017 - val_acc: 0.9711            

Epoch 3/30                                                                          
 - 15s - loss: 0.2140 - acc: 0.9342 - val_loss: 0.1034 - val_acc: 0.9669            

Epoch 4/30                                                                          
 - 15s - loss: 0.1736 - acc: 0.9469 - val_loss: 0.0713 - val_acc: 0.9779            

Epoch 5/30                                                                          
 - 15s - loss: 0.1433 - acc: 0.9551 - val_loss: 0.0490 - val_acc: 0.9840            

Epoch 6/30                                                                          
 - 15s - loss: 0.1321 - acc: 0.9592 - val_loss: 0.0528 - val_acc: 0.9836            

Epoch 7/30                                                                          
 - 15s - loss: 0.1178 - acc: 0.9632 - val_loss: 0.0466 - val_acc: 0.9855            

Epoch 8/30                                                                          
 - 16s - loss: 0.1069 - acc: 0.9661 - val_loss: 0.0466 - val_acc: 0.9851            

Epoch 9/30                                                                          
 - 16s - loss: 0.1033 - acc: 0.9682 - val_loss: 0.0492 - val_acc: 0.9857            

Epoch 10/30                                                                         
 - 16s - loss: 0.0983 - acc: 0.9684 - val_loss: 0.0346 - val_acc: 0.9898            

Epoch 11/30                                                                         
 - 16s - loss: 0.0927 - acc: 0.9702 - val_loss: 0.0362 - val_acc: 0.9880            

Epoch 12/30                                                                         
 - 16s - loss: 0.0929 - acc: 0.9705 - val_loss: 0.0383 - val_acc: 0.9879            

Epoch 13/30                                                                         
 - 16s - loss: 0.0887 - acc: 0.9723 - val_loss: 0.0339 - val_acc: 0.9886            

Epoch 14/30                                                                         
 - 16s - loss: 0.0917 - acc: 0.9710 - val_loss: 0.0369 - val_acc: 0.9870            

Epoch 15/30                                                                         
 - 15s - loss: 0.0918 - acc: 0.9718 - val_loss: 0.0328 - val_acc: 0.9907            

Epoch 16/30                                                                         
 - 16s - loss: 0.0877 - acc: 0.9726 - val_loss: 0.0351 - val_acc: 0.9893            

Epoch 17/30                                                                         
 - 15s - loss: 0.0872 - acc: 0.9728 - val_loss: 0.0370 - val_acc: 0.9880            

Epoch 18/30                                                                         
 - 15s - loss: 0.0857 - acc: 0.9730 - val_loss: 0.0339 - val_acc: 0.9881            

Epoch 1/30                                                                          
 - 34s - loss: 0.3253 - acc: 0.9127 - val_loss: 0.1139 - val_acc: 0.9807            

Epoch 2/30                                                                          
 - 16s - loss: 0.1609 - acc: 0.9636 - val_loss: 0.1716 - val_acc: 0.9705            

Epoch 3/30                                                                          
 - 16s - loss: 0.1301 - acc: 0.9702 - val_loss: 0.1231 - val_acc: 0.9846            

Epoch 4/30                                                                          
 - 16s - loss: 0.0756 - acc: 0.9826 - val_loss: 0.0505 - val_acc: 0.9926            

Epoch 5/30                                                                          
 - 16s - loss: 0.0640 - acc: 0.9851 - val_loss: 0.0426 - val_acc: 0.9929            

Epoch 6/30                                                                          
 - 16s - loss: 0.0590 - acc: 0.9860 - val_loss: 0.0519 - val_acc: 0.9918            

Epoch 7/30                                                                          
 - 16s - loss: 0.0569 - acc: 0.9868 - val_loss: 0.0425 - val_acc: 0.9923            

Epoch 8/30                                                                          
 - 15s - loss: 0.0524 - acc: 0.9878 - val_loss: 0.0678 - val_acc: 0.9919            

Epoch 9/30                                                                          
 - 16s - loss: 0.0520 - acc: 0.9862 - val_loss: 0.0399 - val_acc: 0.9946            

Epoch 10/30                                                                         
 - 15s - loss: 0.0484 - acc: 0.9881 - val_loss: 0.0539 - val_acc: 0.9927            

Epoch 11/30                                                                         
 - 16s - loss: 0.0552 - acc: 0.9874 - val_loss: 0.1110 - val_acc: 0.9887            

Epoch 12/30                                                                         
 - 16s - loss: 0.0394 - acc: 0.9910 - val_loss: 0.0405 - val_acc: 0.9943            

Epoch 1/30                                                                          
 - 42s - loss: 0.4217 - acc: 0.8791 - val_loss: 0.0716 - val_acc: 0.9781          

Epoch 2/30                                                                        
 - 23s - loss: 0.1711 - acc: 0.9516 - val_loss: 0.0684 - val_acc: 0.9813          

Epoch 3/30                                                                        
 - 22s - loss: 0.1250 - acc: 0.9640 - val_loss: 0.0454 - val_acc: 0.9890          

Epoch 4/30                                                                        
 - 23s - loss: 0.0998 - acc: 0.9713 - val_loss: 0.0379 - val_acc: 0.9896          

Epoch 5/30                                                                        
 - 23s - loss: 0.0919 - acc: 0.9740 - val_loss: 0.0381 - val_acc: 0.9895          

Epoch 6/30                                                                        
 - 22s - loss: 0.0796 - acc: 0.9764 - val_loss: 0.0550 - val_acc: 0.9868          

Epoch 7/30                                                                        
 - 23s - loss: 0.0651 - acc: 0.9806 - val_loss: 0.0218 - val_acc: 0.9927          

Epoch 8/30                                                                        
 - 23s - loss: 0.0604 - acc: 0.9813 - val_loss: 0.0271 - val_acc: 0.9925          

Epoch 9/30                                                                        
 - 23s - loss: 0.0575 - acc: 0.9834 - val_loss: 0.0210 - val_acc: 0.9937          

Epoch 10/30                                                                       
 - 23s - loss: 0.0527 - acc: 0.9841 - val_loss: 0.0249 - val_acc: 0.9926          

Epoch 11/30                                                                       
 - 23s - loss: 0.0547 - acc: 0.9838 - val_loss: 0.0197 - val_acc: 0.9944          

Epoch 12/30                                                                       
 - 23s - loss: 0.0532 - acc: 0.9845 - val_loss: 0.0218 - val_acc: 0.9931          

Epoch 13/30                                                                       
 - 23s - loss: 0.0510 - acc: 0.9850 - val_loss: 0.0221 - val_acc: 0.9933          

Epoch 14/30                                                                       
 - 23s - loss: 0.0495 - acc: 0.9854 - val_loss: 0.0172 - val_acc: 0.9945          

Epoch 15/30                                                                       
 - 22s - loss: 0.0440 - acc: 0.9864 - val_loss: 0.0259 - val_acc: 0.9931          

Epoch 16/30                                                                       
 - 22s - loss: 0.0443 - acc: 0.9867 - val_loss: 0.0204 - val_acc: 0.9942          

Epoch 17/30                                                                       
 - 23s - loss: 0.0444 - acc: 0.9861 - val_loss: 0.0193 - val_acc: 0.9939          

Epoch 1/30                                                                        
 - 36s - loss: 0.2182 - acc: 0.9395 - val_loss: 0.1395 - val_acc: 0.9727          

Epoch 2/30                                                                        
 - 16s - loss: 0.1010 - acc: 0.9735 - val_loss: 0.0986 - val_acc: 0.9783          

Epoch 3/30                                                                        
 - 16s - loss: 0.0796 - acc: 0.9790 - val_loss: 0.0588 - val_acc: 0.9870          

Epoch 4/30                                                                        
 - 16s - loss: 0.0678 - acc: 0.9816 - val_loss: 0.0455 - val_acc: 0.9890          

Epoch 5/30                                                                        
 - 16s - loss: 0.0579 - acc: 0.9847 - val_loss: 0.0470 - val_acc: 0.9877          

Epoch 6/30                                                                        
 - 16s - loss: 0.0535 - acc: 0.9848 - val_loss: 0.0567 - val_acc: 0.9885          

Epoch 7/30                                                                        
 - 16s - loss: 0.0287 - acc: 0.9919 - val_loss: 0.0252 - val_acc: 0.9933          

Epoch 8/30                                                                        
 - 16s - loss: 0.0208 - acc: 0.9937 - val_loss: 0.0293 - val_acc: 0.9921          

Epoch 9/30                                                                        
 - 16s - loss: 0.0186 - acc: 0.9941 - val_loss: 0.0225 - val_acc: 0.9939          

Epoch 10/30                                                                       
 - 15s - loss: 0.0172 - acc: 0.9946 - val_loss: 0.0263 - val_acc: 0.9931          

Epoch 11/30                                                                       
 - 15s - loss: 0.0186 - acc: 0.9947 - val_loss: 0.0269 - val_acc: 0.9936          

Epoch 12/30                                                                       
 - 16s - loss: 0.0154 - acc: 0.9954 - val_loss: 0.0224 - val_acc: 0.9936          

Epoch 13/30                                                                       
 - 15s - loss: 0.0117 - acc: 0.9965 - val_loss: 0.0211 - val_acc: 0.9951          

Epoch 14/30                                                                       
 - 15s - loss: 0.0112 - acc: 0.9964 - val_loss: 0.0181 - val_acc: 0.9948          

Epoch 15/30                                                                       
 - 16s - loss: 0.0089 - acc: 0.9974 - val_loss: 0.0215 - val_acc: 0.9949          

Epoch 16/30                                                                       
 - 15s - loss: 0.0107 - acc: 0.9968 - val_loss: 0.0237 - val_acc: 0.9935          

Epoch 17/30                                                                       
 - 16s - loss: 0.0085 - acc: 0.9974 - val_loss: 0.0194 - val_acc: 0.9950          

Epoch 1/30                                                                        
 - 37s - loss: 0.2018 - acc: 0.9386 - val_loss: 0.0844 - val_acc: 0.9777          

Epoch 2/30                                                                        
 - 16s - loss: 0.0913 - acc: 0.9749 - val_loss: 0.0747 - val_acc: 0.9832          

Epoch 3/30                                                                        
 - 16s - loss: 0.0716 - acc: 0.9798 - val_loss: 0.0752 - val_acc: 0.9838          

Epoch 4/30                                                                        
 - 16s - loss: 0.0651 - acc: 0.9826 - val_loss: 0.0730 - val_acc: 0.9824          

Epoch 5/30                                                                        
 - 16s - loss: 0.0517 - acc: 0.9859 - val_loss: 0.0763 - val_acc: 0.9854          

Epoch 6/30                                                                        
 - 16s - loss: 0.0506 - acc: 0.9851 - val_loss: 0.0316 - val_acc: 0.9921          

Epoch 7/30                                                                        
 - 16s - loss: 0.0440 - acc: 0.9865 - val_loss: 0.0342 - val_acc: 0.9931          

Epoch 8/30                                                                        
 - 16s - loss: 0.0436 - acc: 0.9879 - val_loss: 0.0499 - val_acc: 0.9898          

Epoch 9/30                                                                        
 - 16s - loss: 0.0228 - acc: 0.9930 - val_loss: 0.0273 - val_acc: 0.9933          

Epoch 10/30                                                                       
 - 16s - loss: 0.0182 - acc: 0.9946 - val_loss: 0.0246 - val_acc: 0.9939          

Epoch 11/30                                                                       
 - 15s - loss: 0.0171 - acc: 0.9944 - val_loss: 0.0216 - val_acc: 0.9943          

Epoch 12/30                                                                       
 - 16s - loss: 0.0166 - acc: 0.9949 - val_loss: 0.0220 - val_acc: 0.9943          

Epoch 13/30                                                                       
 - 16s - loss: 0.0144 - acc: 0.9955 - val_loss: 0.0172 - val_acc: 0.9956          

Epoch 14/30                                                                       
 - 15s - loss: 0.0149 - acc: 0.9953 - val_loss: 0.0246 - val_acc: 0.9944          

Epoch 15/30                                                                       
 - 16s - loss: 0.0143 - acc: 0.9954 - val_loss: 0.0202 - val_acc: 0.9949          

Epoch 16/30                                                                       
 - 16s - loss: 0.0102 - acc: 0.9965 - val_loss: 0.0174 - val_acc: 0.9965          

Epoch 1/30                                                                        
 - 36s - loss: 0.3440 - acc: 0.9105 - val_loss: 0.4647 - val_acc: 0.9112          

Epoch 2/30                                                                        
 - 16s - loss: 0.1564 - acc: 0.9628 - val_loss: 0.1167 - val_acc: 0.9768          

Epoch 3/30                                                                        
 - 16s - loss: 0.1286 - acc: 0.9695 - val_loss: 0.0946 - val_acc: 0.9802          

Epoch 4/30                                                                        
 - 15s - loss: 0.0929 - acc: 0.9762 - val_loss: 0.0798 - val_acc: 0.9848          

Epoch 5/30                                                                        
 - 15s - loss: 0.0773 - acc: 0.9798 - val_loss: 0.0497 - val_acc: 0.9870          

Epoch 6/30                                                                        
 - 15s - loss: 0.0614 - acc: 0.9829 - val_loss: 0.0611 - val_acc: 0.9844          

Epoch 7/30                                                                        
 - 15s - loss: 0.0613 - acc: 0.9824 - val_loss: 0.0538 - val_acc: 0.9862          

Epoch 8/30                                                                        
 - 16s - loss: 0.0354 - acc: 0.9898 - val_loss: 0.0391 - val_acc: 0.9902          

Epoch 9/30                                                                        
 - 16s - loss: 0.0311 - acc: 0.9905 - val_loss: 0.0272 - val_acc: 0.9924          

Epoch 10/30                                                                       
 - 16s - loss: 0.0286 - acc: 0.9916 - val_loss: 0.0263 - val_acc: 0.9927          

Epoch 11/30                                                                       
 - 16s - loss: 0.0251 - acc: 0.9916 - val_loss: 0.0282 - val_acc: 0.9927          

Epoch 12/30                                                                       
 - 16s - loss: 0.0248 - acc: 0.9929 - val_loss: 0.0327 - val_acc: 0.9914          

Epoch 13/30                                                                       
 - 16s - loss: 0.0187 - acc: 0.9941 - val_loss: 0.0211 - val_acc: 0.9940          

Epoch 14/30                                                                       
 - 16s - loss: 0.0170 - acc: 0.9945 - val_loss: 0.0231 - val_acc: 0.9942          

Epoch 15/30                                                                       
 - 16s - loss: 0.0149 - acc: 0.9956 - val_loss: 0.0218 - val_acc: 0.9949          

Epoch 16/30                                                                       
 - 16s - loss: 0.0146 - acc: 0.9956 - val_loss: 0.0213 - val_acc: 0.9945          

Epoch 1/30                                                                        
 - 37s - loss: 0.1987 - acc: 0.9405 - val_loss: 0.0916 - val_acc: 0.9760          

Epoch 2/30                                                                        
 - 17s - loss: 0.0918 - acc: 0.9741 - val_loss: 0.0592 - val_acc: 0.9842          

Epoch 3/30                                                                        
 - 16s - loss: 0.0702 - acc: 0.9799 - val_loss: 0.0626 - val_acc: 0.9842          

Epoch 4/30                                                                        
 - 16s - loss: 0.0628 - acc: 0.9821 - val_loss: 0.0604 - val_acc: 0.9877          

Epoch 5/30                                                                        
 - 17s - loss: 0.0332 - acc: 0.9906 - val_loss: 0.0287 - val_acc: 0.9917          

Epoch 6/30                                                                        
 - 17s - loss: 0.0255 - acc: 0.9923 - val_loss: 0.0240 - val_acc: 0.9933          

Epoch 7/30                                                                        
 - 16s - loss: 0.0227 - acc: 0.9931 - val_loss: 0.0218 - val_acc: 0.9935          

Epoch 8/30                                                                        
 - 16s - loss: 0.0210 - acc: 0.9935 - val_loss: 0.0231 - val_acc: 0.9931          

Epoch 9/30                                                                        
 - 16s - loss: 0.0220 - acc: 0.9929 - val_loss: 0.0178 - val_acc: 0.9951          

Epoch 10/30                                                                       
 - 16s - loss: 0.0234 - acc: 0.9927 - val_loss: 0.0226 - val_acc: 0.9948          

Epoch 11/30                                                                       
 - 17s - loss: 0.0198 - acc: 0.9943 - val_loss: 0.0215 - val_acc: 0.9930          

Epoch 12/30                                                                       
 - 16s - loss: 0.0137 - acc: 0.9956 - val_loss: 0.0203 - val_acc: 0.9944          

Epoch 1/30                                                                        
 - 38s - loss: 0.2393 - acc: 0.9237 - val_loss: 0.0820 - val_acc: 0.9819          

Epoch 2/30                                                                        
 - 16s - loss: 0.1017 - acc: 0.9710 - val_loss: 0.0807 - val_acc: 0.9823          

Epoch 3/30                                                                        
 - 17s - loss: 0.0840 - acc: 0.9772 - val_loss: 0.0525 - val_acc: 0.9881          

Epoch 4/30                                                                        
 - 16s - loss: 0.0659 - acc: 0.9811 - val_loss: 0.0464 - val_acc: 0.9873          

Epoch 5/30                                                                        
 - 17s - loss: 0.0676 - acc: 0.9811 - val_loss: 0.0353 - val_acc: 0.9907          

Epoch 6/30                                                                        
 - 16s - loss: 0.0555 - acc: 0.9842 - val_loss: 0.0233 - val_acc: 0.9931          

Epoch 7/30                                                                        
 - 16s - loss: 0.0519 - acc: 0.9843 - val_loss: 0.0322 - val_acc: 0.9917          

Epoch 8/30                                                                        
 - 16s - loss: 0.0477 - acc: 0.9861 - val_loss: 0.0421 - val_acc: 0.9889          

Epoch 9/30                                                                        
 - 17s - loss: 0.0280 - acc: 0.9916 - val_loss: 0.0231 - val_acc: 0.9944          

Epoch 10/30                                                                       
 - 16s - loss: 0.0217 - acc: 0.9936 - val_loss: 0.0180 - val_acc: 0.9942          

Epoch 11/30                                                                       
 - 16s - loss: 0.0215 - acc: 0.9934 - val_loss: 0.0267 - val_acc: 0.9932          

Epoch 12/30                                                                       
 - 16s - loss: 0.0185 - acc: 0.9938 - val_loss: 0.0175 - val_acc: 0.9946          

Epoch 13/30                                                                       
 - 16s - loss: 0.0199 - acc: 0.9943 - val_loss: 0.0219 - val_acc: 0.9937          

Epoch 14/30                                                                       
 - 16s - loss: 0.0207 - acc: 0.9930 - val_loss: 0.0243 - val_acc: 0.9939          

Epoch 15/30                                                                       
 - 15s - loss: 0.0154 - acc: 0.9959 - val_loss: 0.0240 - val_acc: 0.9937          

Epoch 1/30                                                                        
 - 38s - loss: 0.3076 - acc: 0.9264 - val_loss: 0.2360 - val_acc: 0.9562          

Epoch 2/30                                                                        
 - 17s - loss: 0.1637 - acc: 0.9674 - val_loss: 0.1731 - val_acc: 0.9718          

Epoch 3/30                                                                        
 - 17s - loss: 0.1179 - acc: 0.9746 - val_loss: 0.0739 - val_acc: 0.9862          

Epoch 4/30                                                                        
 - 16s - loss: 0.0817 - acc: 0.9808 - val_loss: 0.0571 - val_acc: 0.9871          

Epoch 5/30                                                                        
 - 17s - loss: 0.0611 - acc: 0.9843 - val_loss: 0.0501 - val_acc: 0.9892          

Epoch 6/30                                                                        
 - 16s - loss: 0.0638 - acc: 0.9838 - val_loss: 0.0423 - val_acc: 0.9915          

Epoch 7/30                                                                        
 - 17s - loss: 0.0566 - acc: 0.9853 - val_loss: 0.0532 - val_acc: 0.9874          

Epoch 8/30                                                                        
 - 16s - loss: 0.0516 - acc: 0.9860 - val_loss: 0.0610 - val_acc: 0.9835          

Epoch 9/30                                                                        
 - 16s - loss: 0.0295 - acc: 0.9913 - val_loss: 0.0217 - val_acc: 0.9945          

Epoch 10/30                                                                       
 - 16s - loss: 0.0208 - acc: 0.9939 - val_loss: 0.0279 - val_acc: 0.9933          

Epoch 11/30                                                                       
 - 17s - loss: 0.0208 - acc: 0.9935 - val_loss: 0.0295 - val_acc: 0.9929          

Epoch 12/30                                                                       
 - 17s - loss: 0.0174 - acc: 0.9949 - val_loss: 0.0237 - val_acc: 0.9946          

Epoch 1/30                                                                        
 - 38s - loss: 0.1528 - acc: 0.9523 - val_loss: 0.0718 - val_acc: 0.9798          

Epoch 2/30                                                                        
 - 17s - loss: 0.0783 - acc: 0.9770 - val_loss: 0.1404 - val_acc: 0.9625          

Epoch 3/30                                                                        
 - 16s - loss: 0.0572 - acc: 0.9821 - val_loss: 0.0444 - val_acc: 0.9885          

Epoch 4/30                                                                        
 - 16s - loss: 0.0511 - acc: 0.9845 - val_loss: 0.0783 - val_acc: 0.9819          

Epoch 5/30                                                                        
 - 17s - loss: 0.0486 - acc: 0.9858 - val_loss: 0.0327 - val_acc: 0.9906          

Epoch 6/30                                                                        
 - 16s - loss: 0.0443 - acc: 0.9870 - val_loss: 0.0463 - val_acc: 0.9881          

Epoch 7/30                                                                        
 - 17s - loss: 0.0375 - acc: 0.9888 - val_loss: 0.0380 - val_acc: 0.9913          

Epoch 8/30                                                                        
 - 17s - loss: 0.0223 - acc: 0.9933 - val_loss: 0.0181 - val_acc: 0.9950          

Epoch 9/30                                                                        
 - 16s - loss: 0.0164 - acc: 0.9950 - val_loss: 0.0216 - val_acc: 0.9946          

Epoch 10/30                                                                       
 - 16s - loss: 0.0158 - acc: 0.9949 - val_loss: 0.0203 - val_acc: 0.9949          

Epoch 11/30                                                                       
 - 16s - loss: 0.0126 - acc: 0.9961 - val_loss: 0.0155 - val_acc: 0.9958          

Epoch 12/30                                                                       
 - 17s - loss: 0.0109 - acc: 0.9966 - val_loss: 0.0182 - val_acc: 0.9963          

Epoch 13/30                                                                       
 - 16s - loss: 0.0099 - acc: 0.9971 - val_loss: 0.0135 - val_acc: 0.9961          

Epoch 14/30                                                                       
 - 16s - loss: 0.0096 - acc: 0.9976 - val_loss: 0.0199 - val_acc: 0.9949          

Epoch 15/30                                                                       
 - 16s - loss: 0.0089 - acc: 0.9970 - val_loss: 0.0183 - val_acc: 0.9956          

Epoch 16/30                                                                       
 - 16s - loss: 0.0085 - acc: 0.9977 - val_loss: 0.0140 - val_acc: 0.9957          

Epoch 1/30                                                                        
 - 39s - loss: 0.3061 - acc: 0.9002 - val_loss: 0.0601 - val_acc: 0.9814          

Epoch 2/30                                                                        
 - 17s - loss: 0.1151 - acc: 0.9649 - val_loss: 0.1378 - val_acc: 0.9632          

Epoch 3/30                                                                        
 - 17s - loss: 0.0940 - acc: 0.9704 - val_loss: 0.0445 - val_acc: 0.9855          

Epoch 4/30                                                                        
 - 17s - loss: 0.0835 - acc: 0.9745 - val_loss: 0.0398 - val_acc: 0.9886          

Epoch 5/30                                                                        
 - 16s - loss: 0.0738 - acc: 0.9776 - val_loss: 0.0346 - val_acc: 0.9904          

Epoch 6/30                                                                        
 - 16s - loss: 0.0686 - acc: 0.9790 - val_loss: 0.0288 - val_acc: 0.9908          

Epoch 7/30                                                                        
 - 17s - loss: 0.0665 - acc: 0.9799 - val_loss: 0.0305 - val_acc: 0.9919          

Epoch 8/30                                                                        
 - 16s - loss: 0.0599 - acc: 0.9820 - val_loss: 0.0310 - val_acc: 0.9918          

Epoch 9/30                                                                        
 - 17s - loss: 0.0444 - acc: 0.9871 - val_loss: 0.0204 - val_acc: 0.9942          

Epoch 10/30                                                                       
 - 16s - loss: 0.0370 - acc: 0.9885 - val_loss: 0.0171 - val_acc: 0.9937          

Epoch 11/30                                                                       
 - 16s - loss: 0.0323 - acc: 0.9899 - val_loss: 0.0181 - val_acc: 0.9942          

Epoch 12/30                                                                       
 - 17s - loss: 0.0312 - acc: 0.9906 - val_loss: 0.0221 - val_acc: 0.9927          

Epoch 13/30                                                                       
 - 17s - loss: 0.0324 - acc: 0.9901 - val_loss: 0.0166 - val_acc: 0.9948          

Epoch 14/30                                                                       
 - 16s - loss: 0.0268 - acc: 0.9916 - val_loss: 0.0160 - val_acc: 0.9949          

Epoch 15/30                                                                       
 - 16s - loss: 0.0264 - acc: 0.9919 - val_loss: 0.0179 - val_acc: 0.9950          

Epoch 16/30                                                                       
 - 16s - loss: 0.0229 - acc: 0.9923 - val_loss: 0.0178 - val_acc: 0.9937          

Epoch 17/30                                                                       
 - 17s - loss: 0.0245 - acc: 0.9929 - val_loss: 0.0151 - val_acc: 0.9952          

Epoch 18/30                                                                       
 - 17s - loss: 0.0237 - acc: 0.9921 - val_loss: 0.0146 - val_acc: 0.9948          

Epoch 19/30                                                                       
 - 17s - loss: 0.0245 - acc: 0.9922 - val_loss: 0.0157 - val_acc: 0.9951          

Epoch 20/30                                                                       
 - 18s - loss: 0.0217 - acc: 0.9937 - val_loss: 0.0159 - val_acc: 0.9946          

Epoch 21/30                                                                       
 - 17s - loss: 0.0225 - acc: 0.9928 - val_loss: 0.0162 - val_acc: 0.9944          

Epoch 1/30                                                                        
 - 40s - loss: 0.2175 - acc: 0.9377 - val_loss: 0.1437 - val_acc: 0.9685          

Epoch 2/30                                                                        
 - 18s - loss: 0.1121 - acc: 0.9708 - val_loss: 0.1757 - val_acc: 0.9469          

Epoch 3/30                                                                        
 - 16s - loss: 0.0851 - acc: 0.9763 - val_loss: 0.1054 - val_acc: 0.9760          

Epoch 4/30                                                                        
 - 17s - loss: 0.0679 - acc: 0.9810 - val_loss: 0.0502 - val_acc: 0.9879          

Epoch 5/30                                                                        
 - 17s - loss: 0.0605 - acc: 0.9834 - val_loss: 0.0735 - val_acc: 0.9821          

Epoch 6/30                                                                        
 - 18s - loss: 0.0534 - acc: 0.9850 - val_loss: 0.0455 - val_acc: 0.9892          

Epoch 7/30                                                                        
 - 17s - loss: 0.0467 - acc: 0.9865 - val_loss: 0.0520 - val_acc: 0.9864          

Epoch 8/30                                                                        
 - 17s - loss: 0.0438 - acc: 0.9871 - val_loss: 0.0477 - val_acc: 0.9864          

Epoch 9/30                                                                        
 - 18s - loss: 0.0246 - acc: 0.9930 - val_loss: 0.0227 - val_acc: 0.9948          

Epoch 10/30                                                                       
 - 17s - loss: 0.0193 - acc: 0.9943 - val_loss: 0.0225 - val_acc: 0.9942          

Epoch 11/30                                                                       
 - 17s - loss: 0.0183 - acc: 0.9943 - val_loss: 0.0257 - val_acc: 0.9937          

Epoch 12/30                                                                       
 - 17s - loss: 0.0184 - acc: 0.9943 - val_loss: 0.0254 - val_acc: 0.9936          

Epoch 13/30                                                                       
 - 17s - loss: 0.0129 - acc: 0.9961 - val_loss: 0.0204 - val_acc: 0.9949          

Epoch 14/30                                                                       
 - 18s - loss: 0.0119 - acc: 0.9965 - val_loss: 0.0203 - val_acc: 0.9945          

Epoch 15/30                                                                       
 - 18s - loss: 0.0113 - acc: 0.9970 - val_loss: 0.0206 - val_acc: 0.9950          

Epoch 16/30                                                                       
 - 18s - loss: 0.0081 - acc: 0.9976 - val_loss: 0.0200 - val_acc: 0.9948          

Epoch 17/30                                                                       
 - 17s - loss: 0.0084 - acc: 0.9970 - val_loss: 0.0199 - val_acc: 0.9946          

Epoch 18/30                                                                       
 - 17s - loss: 0.0087 - acc: 0.9973 - val_loss: 0.0194 - val_acc: 0.9954          

Epoch 19/30                                                                       
 - 17s - loss: 0.0085 - acc: 0.9975 - val_loss: 0.0193 - val_acc: 0.9957          

Epoch 20/30                                                                       
 - 16s - loss: 0.0074 - acc: 0.9976 - val_loss: 0.0202 - val_acc: 0.9949          

Epoch 21/30                                                                       
 - 16s - loss: 0.0082 - acc: 0.9974 - val_loss: 0.0209 - val_acc: 0.9945          

Epoch 22/30                                                                       
 - 17s - loss: 0.0077 - acc: 0.9975 - val_loss: 0.0208 - val_acc: 0.9945          

Epoch 1/30                                                                        
 - 41s - loss: 0.1596 - acc: 0.9493 - val_loss: 0.0771 - val_acc: 0.9799          

Epoch 2/30                                                                        
 - 17s - loss: 0.0727 - acc: 0.9786 - val_loss: 0.0966 - val_acc: 0.9786          

Epoch 3/30                                                                        
 - 17s - loss: 0.0607 - acc: 0.9819 - val_loss: 0.0732 - val_acc: 0.9810          

Epoch 4/30                                                                        
 - 17s - loss: 0.0499 - acc: 0.9849 - val_loss: 0.0421 - val_acc: 0.9876          

Epoch 5/30                                                                        
 - 17s - loss: 0.0465 - acc: 0.9862 - val_loss: 0.0570 - val_acc: 0.9860          

Epoch 6/30                                                                        
 - 16s - loss: 0.0430 - acc: 0.9868 - val_loss: 0.0492 - val_acc: 0.9876          

Epoch 7/30                                                                        
 - 17s - loss: 0.0224 - acc: 0.9933 - val_loss: 0.0226 - val_acc: 0.9942          

Epoch 8/30                                                                        
 - 17s - loss: 0.0176 - acc: 0.9945 - val_loss: 0.0245 - val_acc: 0.9936          

Epoch 9/30                                                                        
 - 17s - loss: 0.0169 - acc: 0.9953 - val_loss: 0.0185 - val_acc: 0.9950          

Epoch 10/30                                                                       
 - 16s - loss: 0.0152 - acc: 0.9956 - val_loss: 0.0251 - val_acc: 0.9937          

Epoch 11/30                                                                       
 - 17s - loss: 0.0142 - acc: 0.9958 - val_loss: 0.0183 - val_acc: 0.9950          

Epoch 12/30                                                                       
 - 18s - loss: 0.0151 - acc: 0.9949 - val_loss: 0.0211 - val_acc: 0.9943          

Epoch 13/30                                                                       
 - 18s - loss: 0.0148 - acc: 0.9950 - val_loss: 0.0220 - val_acc: 0.9945          

Epoch 14/30                                                                       
 - 16s - loss: 0.0100 - acc: 0.9968 - val_loss: 0.0175 - val_acc: 0.9960          

Epoch 15/30                                                                       
 - 17s - loss: 0.0089 - acc: 0.9973 - val_loss: 0.0173 - val_acc: 0.9961          

Epoch 16/30                                                                       
 - 17s - loss: 0.0077 - acc: 0.9975 - val_loss: 0.0191 - val_acc: 0.9949          

Epoch 17/30                                                                       
 - 17s - loss: 0.0086 - acc: 0.9976 - val_loss: 0.0167 - val_acc: 0.9958          

Epoch 18/30                                                                       
 - 17s - loss: 0.0075 - acc: 0.9976 - val_loss: 0.0179 - val_acc: 0.9956          

Epoch 19/30                                                                       
 - 17s - loss: 0.0081 - acc: 0.9976 - val_loss: 0.0170 - val_acc: 0.9943          

Epoch 20/30                                                                       
 - 17s - loss: 0.0080 - acc: 0.9976 - val_loss: 0.0156 - val_acc: 0.9952          

Epoch 21/30                                                                       
 - 16s - loss: 0.0064 - acc: 0.9980 - val_loss: 0.0139 - val_acc: 0.9960          

Epoch 22/30                                                                       
 - 17s - loss: 0.0058 - acc: 0.9982 - val_loss: 0.0135 - val_acc: 0.9955          

Epoch 23/30                                                                       
 - 18s - loss: 0.0050 - acc: 0.9987 - val_loss: 0.0144 - val_acc: 0.9954          

Epoch 24/30                                                                       
 - 18s - loss: 0.0054 - acc: 0.9984 - val_loss: 0.0162 - val_acc: 0.9951          

Epoch 25/30                                                                       
 - 17s - loss: 0.0055 - acc: 0.9984 - val_loss: 0.0135 - val_acc: 0.9962          

100%|██████████| 50/50 [3:55:42<00:00, 378.77s/it, best loss: -0.9966666666666667]

















In [12]:



    


best_params = space_eval(space, best)
print('best hyper params: \n', best_params)



    


















    






best hyper params: 
 {'conv1': 128, 'conv2': 128, 'conv3': 32, 'conv4': 32, 'dense1': 512, 'dropout1': 0, 'dropout2': 0, 'dropout3': 0, 'kernel_size_1': 5, 'kernel_size_2': 3, 'kernel_size_3': 3, 'kernel_size_4': 5, 'loss': 'kullback_leibler_divergence', 'opt': 'adam', 'pooling_size_1': 3, 'pooling_size_2': 2}

Huấn luyện lại mô hình với bộ tham số tốt nhất ở trên.





In [13]:



    


acc, model, history = train_model(train_gen, valid_gen, best_params)
print("validation accuracy: {}".format(acc))



    


















    






Epoch 1/30
 - 43s - loss: 0.1557 - acc: 0.9516 - val_loss: 0.0764 - val_acc: 0.9782
Epoch 2/30
 - 17s - loss: 0.0708 - acc: 0.9782 - val_loss: 0.0628 - val_acc: 0.9832
Epoch 3/30
 - 17s - loss: 0.0596 - acc: 0.9826 - val_loss: 0.0463 - val_acc: 0.9870
Epoch 4/30
 - 17s - loss: 0.0505 - acc: 0.9849 - val_loss: 0.0571 - val_acc: 0.9842
Epoch 5/30
 - 17s - loss: 0.0459 - acc: 0.9856 - val_loss: 0.0496 - val_acc: 0.9873
Epoch 6/30
 - 17s - loss: 0.0247 - acc: 0.9927 - val_loss: 0.0215 - val_acc: 0.9943
Epoch 7/30
 - 17s - loss: 0.0196 - acc: 0.9938 - val_loss: 0.0195 - val_acc: 0.9942
Epoch 8/30
 - 17s - loss: 0.0161 - acc: 0.9953 - val_loss: 0.0248 - val_acc: 0.9943
Epoch 9/30
 - 17s - loss: 0.0170 - acc: 0.9946 - val_loss: 0.0221 - val_acc: 0.9946
Epoch 10/30
 - 17s - loss: 0.0130 - acc: 0.9959 - val_loss: 0.0168 - val_acc: 0.9958
Epoch 11/30
 - 17s - loss: 0.0115 - acc: 0.9970 - val_loss: 0.0195 - val_acc: 0.9951
Epoch 12/30
 - 17s - loss: 0.0109 - acc: 0.9966 - val_loss: 0.0165 - val_acc: 0.9958
Epoch 13/30
 - 17s - loss: 0.0114 - acc: 0.9960 - val_loss: 0.0211 - val_acc: 0.9943
Epoch 14/30
 - 17s - loss: 0.0099 - acc: 0.9969 - val_loss: 0.0191 - val_acc: 0.9945
Epoch 15/30
 - 17s - loss: 0.0095 - acc: 0.9969 - val_loss: 0.0165 - val_acc: 0.9954
validation accuracy: 0.9936904761904762

Kết quả trên tập validation khá cao với acc > 99%

6. So sánh optimizers và loss

6.1 So sánh các optimzers

Mục tiêu của quá trình huấn luyện mô hình ML là giảm độ lỗi của hàm loss function được tính bằng sự khác biệt của giá trị mô hình dự đoán và giá trị thực tế. Để đạt được mục đích này chúng ta thường sử dụng gradient descent. Gradient descent sẽ cập nhật trọng số của mô hình ngược với chiều gradient để giảm độ lỗi của loss function.

Chúng ta sử thường sử dụng 3 optimzer phổ biến sau là adam, sgd, rmsprop để cập nhật trọng số của mô hình. Stochastic Gradient Descent là một biến thể của Gradient Descent, yêu cầu chúng ta phải shuffle dự liệu trước khi huấn luyện. Trong khi đó RMSProp và Adam là 2 optimizer hướng đến việc điều chỉnh learning rate tự động theo quá trình học.

RMSprop (Root mean square propagation) được giới thiệu bởi Geoffrey Hinton. RMSProp giải quyết vấn đề giảm dần learning rate của Adagrad bằng cách chuẩn hóa learning với gradient gần với thời điểm cập nhật mà thôi. Để làm được điều này tác giả chia learning rate cho tổng bình phương gradient giảm dần.

Adam là optimizer phổ biến nhất tại thời điểm hiện tại. Adam cũng tính learning riêng biệt cho từng tham số, tương tự như RMSProp và Adagrad. Adam chuẩn hóa learning của mỗi tham số bằng first và second order moment của gradient.





In [14]:



    


optimizers = ['rmsprop', 'sgd', 'adam']
hists = []
params = best_params
for optimizer in optimizers:
    params['opt'] = optimizer
    print("Train with optimizer: {}".format(optimizer))
    _, _, history = train_model(train_gen, valid_gen, params)
    hists.append((optimizer, history))



    


















    






Train with optimizer: rmsprop
Epoch 1/30
 - 42s - loss: 0.1667 - acc: 0.9488 - val_loss: 0.1342 - val_acc: 0.9690
Epoch 2/30
 - 17s - loss: 0.0814 - acc: 0.9774 - val_loss: 0.0725 - val_acc: 0.9790
Epoch 3/30
 - 17s - loss: 0.0648 - acc: 0.9825 - val_loss: 0.0563 - val_acc: 0.9851
Epoch 4/30
 - 17s - loss: 0.0563 - acc: 0.9844 - val_loss: 0.0644 - val_acc: 0.9887
Epoch 5/30
 - 16s - loss: 0.0484 - acc: 0.9871 - val_loss: 0.0777 - val_acc: 0.9849
Epoch 6/30
 - 18s - loss: 0.0305 - acc: 0.9923 - val_loss: 0.0277 - val_acc: 0.9951
Epoch 7/30
 - 18s - loss: 0.0225 - acc: 0.9941 - val_loss: 0.0271 - val_acc: 0.9939
Epoch 8/30
 - 17s - loss: 0.0184 - acc: 0.9947 - val_loss: 0.0242 - val_acc: 0.9958
Epoch 9/30
 - 17s - loss: 0.0200 - acc: 0.9948 - val_loss: 0.0264 - val_acc: 0.9951
Epoch 10/30
 - 17s - loss: 0.0158 - acc: 0.9956 - val_loss: 0.0263 - val_acc: 0.9955
Epoch 11/30
 - 17s - loss: 0.0149 - acc: 0.9960 - val_loss: 0.0279 - val_acc: 0.9943
Train with optimizer: sgd
Epoch 1/30
 - 42s - loss: 0.2978 - acc: 0.9222 - val_loss: 0.0897 - val_acc: 0.9752
Epoch 2/30
 - 17s - loss: 0.1012 - acc: 0.9732 - val_loss: 0.0592 - val_acc: 0.9830
Epoch 3/30
 - 17s - loss: 0.0730 - acc: 0.9799 - val_loss: 0.0516 - val_acc: 0.9840
Epoch 4/30
 - 17s - loss: 0.0620 - acc: 0.9826 - val_loss: 0.0544 - val_acc: 0.9843
Epoch 5/30
 - 17s - loss: 0.0546 - acc: 0.9846 - val_loss: 0.0456 - val_acc: 0.9865
Epoch 6/30
 - 18s - loss: 0.0496 - acc: 0.9852 - val_loss: 0.0383 - val_acc: 0.9893
Epoch 7/30
 - 17s - loss: 0.0441 - acc: 0.9869 - val_loss: 0.0514 - val_acc: 0.9852
Epoch 8/30
 - 18s - loss: 0.0408 - acc: 0.9876 - val_loss: 0.0281 - val_acc: 0.9914
Epoch 9/30
 - 17s - loss: 0.0385 - acc: 0.9888 - val_loss: 0.0301 - val_acc: 0.9919
Epoch 10/30
 - 17s - loss: 0.0381 - acc: 0.9892 - val_loss: 0.0300 - val_acc: 0.9923
Epoch 11/30
 - 18s - loss: 0.0328 - acc: 0.9906 - val_loss: 0.0287 - val_acc: 0.9924
Train with optimizer: adam
Epoch 1/30
 - 44s - loss: 0.1576 - acc: 0.9501 - val_loss: 0.0492 - val_acc: 0.9856
Epoch 2/30
 - 18s - loss: 0.0708 - acc: 0.9783 - val_loss: 0.0663 - val_acc: 0.9802
Epoch 3/30
 - 17s - loss: 0.0595 - acc: 0.9824 - val_loss: 0.0983 - val_acc: 0.9723
Epoch 4/30
 - 16s - loss: 0.0311 - acc: 0.9902 - val_loss: 0.0226 - val_acc: 0.9924
Epoch 5/30
 - 17s - loss: 0.0239 - acc: 0.9923 - val_loss: 0.0229 - val_acc: 0.9942
Epoch 6/30
 - 16s - loss: 0.0218 - acc: 0.9933 - val_loss: 0.0198 - val_acc: 0.9951
Epoch 7/30
 - 17s - loss: 0.0212 - acc: 0.9933 - val_loss: 0.0159 - val_acc: 0.9957
Epoch 8/30
 - 18s - loss: 0.0213 - acc: 0.9932 - val_loss: 0.0224 - val_acc: 0.9937
Epoch 9/30
 - 17s - loss: 0.0189 - acc: 0.9940 - val_loss: 0.0206 - val_acc: 0.9937
Epoch 10/30
 - 17s - loss: 0.0138 - acc: 0.9958 - val_loss: 0.0133 - val_acc: 0.9952
Epoch 11/30
 - 18s - loss: 0.0132 - acc: 0.9965 - val_loss: 0.0144 - val_acc: 0.9956
Epoch 12/30
 - 18s - loss: 0.0113 - acc: 0.9967 - val_loss: 0.0161 - val_acc: 0.9957
Epoch 13/30
 - 16s - loss: 0.0098 - acc: 0.9969 - val_loss: 0.0170 - val_acc: 0.9951

Plot quá trình huấn luyện mô hình với 3 lọai optimizers khác nhau.





In [15]:



    


for name, history in hists:
    plt.plot(history.history['val_acc'], label=name)

plt.legend(loc='best', shadow=True)
plt.tight_layout()
Chúng ta thấy rằng trong các loại optimzer trên. SGD hội tụ lâu nhất và độ chính xác cũng thấp nhất so với các optimizers còn lại. Trong khi đó, Adam hội tụ nhanh nhất, và có độ chính xác cao nhất.

6.2 So sánh các loss function

Trong bài toán phân loại nhiều lớp. Chúng ta thường sử dụng 2 loại loss function sau:

  • Cross entropy
  • Kullback Leibler Divergence Loss

Cross entropy được sử dụng phổ biến nhất trong bài toán của chúng ta. Cross entropy loss có nền tảng toán học của maximun likelihood được tính bằng tổng của sự khác biệt giữ giá trị dự đoán và giá trị thực tế của dữ liệu. Cross entropy error tốt nhất khi có giá trị bằng 0.

KL loss (Kullback Leibler Divergence Loss) thể hiện sự khác biệt giữ 2 phân bố xác suất. KL loss bằng 0, chứng tỏ 2 phân bố này hoàn toàn giống nhau.

Cross entropy cho bằng toán phân loại nhiều lớn tương đối giống với KL Loss về mặt toán học, nên có thể xem 2 độ lỗi này là một trong bài toán của chúng ta.





In [16]:



    


loss_functions = ['categorical_crossentropy', 'kullback_leibler_divergence']
hists = []
params = best_params
for loss_funct in loss_functions:
    params['loss'] = loss_funct
    print("Train with loss function : {}".format(loss_funct))
    _, _, history = train_model(train_gen, valid_gen, params)
    hists.append((loss_funct, history))



    


















    






Train with loss function : categorical_crossentropy
Epoch 1/30
 - 44s - loss: 0.1543 - acc: 0.9505 - val_loss: 0.0602 - val_acc: 0.9825
Epoch 2/30
 - 17s - loss: 0.0734 - acc: 0.9780 - val_loss: 0.0768 - val_acc: 0.9806
Epoch 3/30
 - 18s - loss: 0.0606 - acc: 0.9826 - val_loss: 0.0422 - val_acc: 0.9885
Epoch 4/30
 - 17s - loss: 0.0506 - acc: 0.9845 - val_loss: 0.0298 - val_acc: 0.9913
Epoch 5/30
 - 18s - loss: 0.0460 - acc: 0.9861 - val_loss: 0.0365 - val_acc: 0.9915
Epoch 6/30
 - 17s - loss: 0.0401 - acc: 0.9883 - val_loss: 0.0471 - val_acc: 0.9885
Epoch 7/30
 - 17s - loss: 0.0239 - acc: 0.9929 - val_loss: 0.0205 - val_acc: 0.9946
Epoch 8/30
 - 18s - loss: 0.0187 - acc: 0.9948 - val_loss: 0.0222 - val_acc: 0.9950
Epoch 9/30
 - 17s - loss: 0.0161 - acc: 0.9948 - val_loss: 0.0215 - val_acc: 0.9939
Epoch 10/30
 - 17s - loss: 0.0136 - acc: 0.9957 - val_loss: 0.0159 - val_acc: 0.9954
Epoch 11/30
 - 18s - loss: 0.0119 - acc: 0.9965 - val_loss: 0.0163 - val_acc: 0.9950
Epoch 12/30
 - 18s - loss: 0.0100 - acc: 0.9968 - val_loss: 0.0157 - val_acc: 0.9957
Epoch 13/30
 - 18s - loss: 0.0109 - acc: 0.9970 - val_loss: 0.0156 - val_acc: 0.9961
Epoch 14/30
 - 18s - loss: 0.0089 - acc: 0.9971 - val_loss: 0.0160 - val_acc: 0.9952
Epoch 15/30
 - 18s - loss: 0.0098 - acc: 0.9969 - val_loss: 0.0158 - val_acc: 0.9955
Epoch 16/30
 - 17s - loss: 0.0088 - acc: 0.9974 - val_loss: 0.0160 - val_acc: 0.9951
Train with loss function : kullback_leibler_divergence
Epoch 1/30
 - 46s - loss: 0.1590 - acc: 0.9510 - val_loss: 0.0868 - val_acc: 0.9736
Epoch 2/30
 - 18s - loss: 0.0743 - acc: 0.9772 - val_loss: 0.0935 - val_acc: 0.9723
Epoch 3/30
 - 18s - loss: 0.0581 - acc: 0.9829 - val_loss: 0.0340 - val_acc: 0.9906
Epoch 4/30
 - 18s - loss: 0.0480 - acc: 0.9854 - val_loss: 0.0650 - val_acc: 0.9839
Epoch 5/30
 - 18s - loss: 0.0446 - acc: 0.9874 - val_loss: 0.0482 - val_acc: 0.9867
Epoch 6/30
 - 17s - loss: 0.0282 - acc: 0.9914 - val_loss: 0.0165 - val_acc: 0.9964
Epoch 7/30
 - 18s - loss: 0.0193 - acc: 0.9938 - val_loss: 0.0253 - val_acc: 0.9937
Epoch 8/30
 - 17s - loss: 0.0185 - acc: 0.9940 - val_loss: 0.0180 - val_acc: 0.9943
Epoch 9/30
 - 17s - loss: 0.0135 - acc: 0.9955 - val_loss: 0.0182 - val_acc: 0.9952

Plot quá trình huấn luyện mô hình với 2 loại loss function khác nhau.





In [17]:



    


for name, history in hists:
    plt.plot(history.history['val_acc'], label=name)

plt.legend(loc='best', shadow=True)
plt.tight_layout()

Chúng ta thấy rằng không có sự khác biệt rõ rằng về tốc độ hội tụ giữ 2 hàm loss function là cross-entropy và KL loss trong bài toán của chúng ta.

7. Đánh giá mô hình.

Chúng ta sẽ xem xét một số lỗi của mô hình dự huấn luyện được. Một số lỗi dễ dàng được phát hiện bằng confusion matrix thể hiện xác xuất/số ảnh bị phân loại nhầm thành lớp khác.





In [18]:



    


def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

# Predict the values from the validation dataset
Y_pred = model.predict(X_val)
# Convert predictions classes to one hot vectors 
Y_pred_classes = np.argmax(Y_pred,axis = 1) 
# Convert validation observations to one hot vectors
Y_true = np.argmax(Y_val,axis = 1) 
# compute the confusion matrix
confusion_mtx = confusion_matrix(Y_true, Y_pred_classes) 
# plot the confusion matrix
plot_confusion_matrix(confusion_mtx, classes = range(10))

Các giá trị trên đường chéo rất cao, chúng ta mô hình chúng ta có độ chính xác rất tốt. Nhìn vào confusion matrix ở trên, chúng ta có một số nhận xét như sau:

  • Số 4 hay nhầm lẫn với số 9, bởi vì khi viết tay đường nét của 2 số này tương tự nhau khá nhiều
  • số 3 và số 8, cũng hay bị tình trạng tương tự.

Hiển thị một số trường hợp bị sai

Để có cái nhìn rõ hơn về một số mẫu bị sai, chúng ta quan sát top các mẫu có giá trị dự đoán khác nhất so với nhãn thật





In [19]:



    


# Display some error results 

# Errors are difference between predicted labels and true labels
errors = (Y_pred_classes - Y_true != 0)

Y_pred_classes_errors = Y_pred_classes[errors]
Y_pred_errors = Y_pred[errors]
Y_true_errors = Y_true[errors]
X_val_errors = X_val[errors]

def display_errors(errors_index,img_errors,pred_errors, obs_errors):
    """ This function shows 6 images with their predicted and real labels"""
    n = 0
    nrows = 2
    ncols = 3
    fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)
    for row in range(nrows):
        for col in range(ncols):
            error = errors_index[n]
            ax[row,col].imshow((img_errors[error]).reshape((28,28)))
            ax[row,col].set_title("Predicted label :{}\nTrue label :{}".format(pred_errors[error],obs_errors[error]))
            n += 1
    fig.tight_layout()
    
# Probabilities of the wrong predicted numbers
Y_pred_errors_prob = np.max(Y_pred_errors,axis = 1)

# Predicted probabilities of the true values in the error set
true_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))

# Difference between the probability of the predicted label and the true label
delta_pred_true_errors = Y_pred_errors_prob - true_prob_errors

# Sorted list of the delta prob errors
sorted_dela_errors = np.argsort(delta_pred_true_errors)

# Top 6 errors 
most_important_errors = sorted_dela_errors[-6:]

# Show the top 6 errors
display_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)

Với các mẫu ảnh sai, chúng ta có thể thấy rằng những mẫu này rất khó nhận dạng nhầm lẫn sáng các lớp khác. ví dụ số 9 và 4 hay là 3 và 8

8. Kfold, Predict và submit kết quả

Chúng ta huấn luyện lại mô hình sử dụng kfold, kết quả cuối dự đoán cuối cùng là trung bình cộng sự đoán của các mô hình huấn luyện trên mỗi fold. Chúng ta chọn nhãn là lớp được dự đoán có xác suất cao nhất mà mô hình nhận dạng được





In [20]:



    


kf = KFold(n_splits=5)
preds = []
for train_index, valid_index in kf.split(X):
    X_train, Y_train, X_val, Y_val = X[train_index], Y[train_index], X[valid_index], Y[valid_index]
    train_gen = train_aug.flow(X_train, Y_train, batch_size=batch_size)
    valid_gen = test_aug.flow(X_val, Y_val, batch_size=batch_size)
    acc, model, history = train_model(train_gen, valid_gen, best_params)
    
    pred = model.predict(X_test)
    preds.append(pred)



    


















    






Epoch 1/30
 - 41s - loss: 0.1605 - acc: 0.9510 - val_loss: 0.1090 - val_acc: 0.9704
Epoch 2/30
 - 18s - loss: 0.0744 - acc: 0.9780 - val_loss: 0.0754 - val_acc: 0.9787
Epoch 3/30
 - 17s - loss: 0.0538 - acc: 0.9839 - val_loss: 0.0673 - val_acc: 0.9818
Epoch 4/30
 - 17s - loss: 0.0577 - acc: 0.9837 - val_loss: 0.0437 - val_acc: 0.9864
Epoch 5/30
 - 17s - loss: 0.0434 - acc: 0.9868 - val_loss: 0.0358 - val_acc: 0.9901
Epoch 6/30
 - 17s - loss: 0.0433 - acc: 0.9873 - val_loss: 0.0448 - val_acc: 0.9877
Epoch 7/30
 - 16s - loss: 0.0363 - acc: 0.9893 - val_loss: 0.0391 - val_acc: 0.9907
Epoch 8/30
 - 17s - loss: 0.0214 - acc: 0.9934 - val_loss: 0.0252 - val_acc: 0.9926
Epoch 9/30
 - 17s - loss: 0.0169 - acc: 0.9946 - val_loss: 0.0249 - val_acc: 0.9927
Epoch 10/30
 - 18s - loss: 0.0145 - acc: 0.9957 - val_loss: 0.0328 - val_acc: 0.9907
Epoch 11/30
 - 17s - loss: 0.0152 - acc: 0.9951 - val_loss: 0.0212 - val_acc: 0.9935
Epoch 12/30
 - 18s - loss: 0.0160 - acc: 0.9950 - val_loss: 0.0283 - val_acc: 0.9925
Epoch 13/30
 - 17s - loss: 0.0128 - acc: 0.9959 - val_loss: 0.0228 - val_acc: 0.9937
Epoch 14/30
 - 18s - loss: 0.0096 - acc: 0.9973 - val_loss: 0.0217 - val_acc: 0.9939
Epoch 1/30
 - 43s - loss: 0.1577 - acc: 0.9516 - val_loss: 0.0594 - val_acc: 0.9850
Epoch 2/30
 - 17s - loss: 0.0745 - acc: 0.9779 - val_loss: 0.0811 - val_acc: 0.9805
Epoch 3/30
 - 18s - loss: 0.0598 - acc: 0.9817 - val_loss: 0.0641 - val_acc: 0.9839
Epoch 4/30
 - 18s - loss: 0.0299 - acc: 0.9910 - val_loss: 0.0282 - val_acc: 0.9912
Epoch 5/30
 - 17s - loss: 0.0229 - acc: 0.9932 - val_loss: 0.0210 - val_acc: 0.9944
Epoch 6/30
 - 18s - loss: 0.0216 - acc: 0.9929 - val_loss: 0.0273 - val_acc: 0.9924
Epoch 7/30
 - 17s - loss: 0.0214 - acc: 0.9935 - val_loss: 0.0249 - val_acc: 0.9929
Epoch 8/30
 - 17s - loss: 0.0148 - acc: 0.9953 - val_loss: 0.0191 - val_acc: 0.9946
Epoch 9/30
 - 18s - loss: 0.0127 - acc: 0.9960 - val_loss: 0.0201 - val_acc: 0.9940
Epoch 10/30
 - 17s - loss: 0.0132 - acc: 0.9963 - val_loss: 0.0197 - val_acc: 0.9945
Epoch 11/30
 - 17s - loss: 0.0113 - acc: 0.9963 - val_loss: 0.0190 - val_acc: 0.9951
Epoch 12/30
 - 18s - loss: 0.0106 - acc: 0.9972 - val_loss: 0.0190 - val_acc: 0.9944
Epoch 13/30
 - 17s - loss: 0.0107 - acc: 0.9967 - val_loss: 0.0185 - val_acc: 0.9949
Epoch 14/30
 - 17s - loss: 0.0107 - acc: 0.9968 - val_loss: 0.0180 - val_acc: 0.9956
Epoch 15/30
 - 18s - loss: 0.0096 - acc: 0.9970 - val_loss: 0.0185 - val_acc: 0.9952
Epoch 16/30
 - 18s - loss: 0.0088 - acc: 0.9974 - val_loss: 0.0182 - val_acc: 0.9955
Epoch 17/30
 - 18s - loss: 0.0089 - acc: 0.9976 - val_loss: 0.0182 - val_acc: 0.9955
Epoch 1/30
 - 47s - loss: 0.1586 - acc: 0.9507 - val_loss: 0.1703 - val_acc: 0.9568
Epoch 2/30
 - 17s - loss: 0.0760 - acc: 0.9769 - val_loss: 0.0712 - val_acc: 0.9842
Epoch 3/30
 - 18s - loss: 0.0575 - acc: 0.9832 - val_loss: 0.0607 - val_acc: 0.9860
Epoch 4/30
 - 19s - loss: 0.0477 - acc: 0.9855 - val_loss: 0.0382 - val_acc: 0.9879
Epoch 5/30
 - 18s - loss: 0.0472 - acc: 0.9859 - val_loss: 0.0358 - val_acc: 0.9902
Epoch 6/30
 - 18s - loss: 0.0426 - acc: 0.9872 - val_loss: 0.0674 - val_acc: 0.9796
Epoch 7/30
 - 18s - loss: 0.0361 - acc: 0.9892 - val_loss: 0.0361 - val_acc: 0.9914
Epoch 8/30
 - 18s - loss: 0.0224 - acc: 0.9935 - val_loss: 0.0275 - val_acc: 0.9939
Epoch 9/30
 - 18s - loss: 0.0164 - acc: 0.9947 - val_loss: 0.0233 - val_acc: 0.9939
Epoch 10/30
 - 17s - loss: 0.0141 - acc: 0.9956 - val_loss: 0.0222 - val_acc: 0.9943
Epoch 11/30
 - 17s - loss: 0.0150 - acc: 0.9954 - val_loss: 0.0254 - val_acc: 0.9932
Epoch 12/30
 - 18s - loss: 0.0141 - acc: 0.9957 - val_loss: 0.0301 - val_acc: 0.9926
Epoch 13/30
 - 18s - loss: 0.0103 - acc: 0.9969 - val_loss: 0.0232 - val_acc: 0.9938
Epoch 1/30
 - 46s - loss: 0.1574 - acc: 0.9497 - val_loss: 0.1697 - val_acc: 0.9568
Epoch 2/30
 - 18s - loss: 0.0715 - acc: 0.9782 - val_loss: 0.0979 - val_acc: 0.9751
Epoch 3/30
 - 17s - loss: 0.0613 - acc: 0.9819 - val_loss: 0.0497 - val_acc: 0.9844
Epoch 4/30
 - 18s - loss: 0.0509 - acc: 0.9849 - val_loss: 0.0696 - val_acc: 0.9800
Epoch 5/30
 - 18s - loss: 0.0426 - acc: 0.9871 - val_loss: 0.0431 - val_acc: 0.9881
Epoch 6/30
 - 18s - loss: 0.0387 - acc: 0.9874 - val_loss: 0.0418 - val_acc: 0.9905
Epoch 7/30
 - 18s - loss: 0.0410 - acc: 0.9879 - val_loss: 0.0347 - val_acc: 0.9915
Epoch 8/30
 - 18s - loss: 0.0382 - acc: 0.9882 - val_loss: 0.0553 - val_acc: 0.9883
Epoch 9/30
 - 18s - loss: 0.0360 - acc: 0.9897 - val_loss: 0.0294 - val_acc: 0.9912
Epoch 10/30
 - 18s - loss: 0.0340 - acc: 0.9899 - val_loss: 0.0279 - val_acc: 0.9920
Epoch 11/30
 - 18s - loss: 0.0294 - acc: 0.9909 - val_loss: 0.0483 - val_acc: 0.9886
Epoch 12/30
 - 18s - loss: 0.0308 - acc: 0.9903 - val_loss: 0.0374 - val_acc: 0.9924
Epoch 13/30
 - 19s - loss: 0.0154 - acc: 0.9952 - val_loss: 0.0220 - val_acc: 0.9943
Epoch 14/30
 - 19s - loss: 0.0110 - acc: 0.9967 - val_loss: 0.0219 - val_acc: 0.9952
Epoch 15/30
 - 18s - loss: 0.0100 - acc: 0.9969 - val_loss: 0.0211 - val_acc: 0.9942
Epoch 16/30
 - 18s - loss: 0.0101 - acc: 0.9969 - val_loss: 0.0230 - val_acc: 0.9945
Epoch 17/30
 - 18s - loss: 0.0116 - acc: 0.9964 - val_loss: 0.0209 - val_acc: 0.9943
Epoch 18/30
 - 18s - loss: 0.0093 - acc: 0.9971 - val_loss: 0.0286 - val_acc: 0.9933
Epoch 19/30
 - 17s - loss: 0.0099 - acc: 0.9967 - val_loss: 0.0191 - val_acc: 0.9940
Epoch 20/30
 - 18s - loss: 0.0115 - acc: 0.9965 - val_loss: 0.0244 - val_acc: 0.9942
Epoch 21/30
 - 19s - loss: 0.0095 - acc: 0.9968 - val_loss: 0.0193 - val_acc: 0.9955
Epoch 22/30
 - 18s - loss: 0.0064 - acc: 0.9979 - val_loss: 0.0182 - val_acc: 0.9951
Epoch 23/30
 - 18s - loss: 0.0065 - acc: 0.9980 - val_loss: 0.0200 - val_acc: 0.9954
Epoch 24/30
 - 18s - loss: 0.0056 - acc: 0.9983 - val_loss: 0.0199 - val_acc: 0.9949
Epoch 25/30
 - 18s - loss: 0.0056 - acc: 0.9982 - val_loss: 0.0193 - val_acc: 0.9952
Epoch 1/30
 - 47s - loss: 0.1594 - acc: 0.9504 - val_loss: 0.0880 - val_acc: 0.9733
Epoch 2/30
 - 18s - loss: 0.0724 - acc: 0.9779 - val_loss: 0.0653 - val_acc: 0.9835
Epoch 3/30
 - 17s - loss: 0.0589 - acc: 0.9827 - val_loss: 0.0463 - val_acc: 0.9896
Epoch 4/30
 - 17s - loss: 0.0504 - acc: 0.9853 - val_loss: 0.0393 - val_acc: 0.9900
Epoch 5/30
 - 17s - loss: 0.0556 - acc: 0.9836 - val_loss: 0.0496 - val_acc: 0.9874
Epoch 6/30
 - 18s - loss: 0.0419 - acc: 0.9879 - val_loss: 0.0629 - val_acc: 0.9830
Epoch 7/30
 - 19s - loss: 0.0230 - acc: 0.9926 - val_loss: 0.0227 - val_acc: 0.9949
Epoch 8/30
 - 18s - loss: 0.0182 - acc: 0.9944 - val_loss: 0.0201 - val_acc: 0.9955
Epoch 9/30
 - 17s - loss: 0.0168 - acc: 0.9945 - val_loss: 0.0234 - val_acc: 0.9944
Epoch 10/30
 - 18s - loss: 0.0159 - acc: 0.9947 - val_loss: 0.0230 - val_acc: 0.9938
Epoch 11/30
 - 18s - loss: 0.0121 - acc: 0.9961 - val_loss: 0.0216 - val_acc: 0.9952

















In [21]:



    


# predict results
results = np.mean(preds, axis=0)
# select the indix with the maximum probability
results = np.argmax(results,axis = 1)

results = pd.Series(results,name="Label")



    











In [22]:



    


submission = pd.concat([pd.Series(range(1,28001),name = "ImageId"),results],axis = 1)

submission.to_csv("cnn_mnist_datagen.csv",index=False)

Lúc chạy thực tế, cần thay max_evals lúc search tham số thành 50 để có được accuracy trên tập test > 0.997. Kết quả dưới đây được submit trên kaggle mà không sử dụng kfold 





In [ ]:

Content source: pbcquoc/pbcquoc.github.io

Similar notebooks:

notebook.community | gallery | about