Accompanying code examples of the book "Introduction to Artificial Neural Networks and Deep Learning: A Practical Guide with Applications in Python" by Sebastian Raschka. All code examples are released under the MIT license. If you find this content useful, please consider supporting the work by buying a copy of the book.

Other code examples and content are available on GitHub. The PDF and ebook versions of the book are available through Leanpub.


In [1]:
%load_ext watermark
%watermark -a 'Sebastian Raschka' -v -p tensorflow


Sebastian Raschka 

CPython 3.6.1
IPython 6.0.0

tensorflow 1.2.0

Model Zoo -- Softmax Regression

Implementation of softmax regression (multinomial logistic regression).


In [2]:
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data


##########################
### DATASET
##########################

mnist = input_data.read_data_sets("./", one_hot=True)


##########################
### SETTINGS
##########################

# Hyperparameters
learning_rate = 0.5
training_epochs = 30
batch_size = 256

# Architecture
n_features = 784
n_classes = 10


##########################
### GRAPH DEFINITION
##########################

g = tf.Graph()
with g.as_default():

    # Input data
    tf_x = tf.placeholder(tf.float32, [None, n_features])
    tf_y = tf.placeholder(tf.float32, [None, n_classes])

    # Model parameters
    params = {
        'weights': tf.Variable(tf.zeros(shape=[n_features, n_classes],
                                               dtype=tf.float32), name='weights'),
        'bias': tf.Variable([[n_classes]], dtype=tf.float32, name='bias')}

    # Softmax regression
    linear = tf.matmul(tf_x, params['weights']) + params['bias']
    pred_proba = tf.nn.softmax(linear, name='predict_probas')
    
    # Loss and optimizer
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
        logits=linear, labels=tf_y), name='cost')
    optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
    train = optimizer.minimize(cost, name='train')

    # Class prediction
    pred_labels = tf.argmax(pred_proba, 1, name='predict_labels')
    correct_prediction = tf.equal(tf.argmax(tf_y, 1), pred_labels)
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='accuracy')

    
##########################
### TRAINING & EVALUATION
##########################

with tf.Session(graph=g) as sess:
    sess.run(tf.global_variables_initializer())

    for epoch in range(training_epochs):
        avg_cost = 0.
        total_batch = mnist.train.num_examples // batch_size

        for i in range(total_batch):
            batch_x, batch_y = mnist.train.next_batch(batch_size)
            _, c = sess.run(['train', 'cost:0'], feed_dict={tf_x: batch_x,
                                                            tf_y: batch_y})
            avg_cost += c
        
        train_acc = sess.run('accuracy:0', feed_dict={tf_x: mnist.train.images,
                                                      tf_y: mnist.train.labels})
        valid_acc = sess.run('accuracy:0', feed_dict={tf_x: mnist.validation.images,
                                                      tf_y: mnist.validation.labels})  
        
        print("Epoch: %03d | AvgCost: %.3f" % (epoch + 1, avg_cost / (i + 1)), end="")
        print(" | Train/Valid ACC: %.3f/%.3f" % (train_acc, valid_acc))
        
    test_acc = sess.run(accuracy, feed_dict={tf_x: mnist.test.images,
                                             tf_y: mnist.test.labels})
    print('Test ACC: %.3f' % test_acc)


Extracting ./train-images-idx3-ubyte.gz
Extracting ./train-labels-idx1-ubyte.gz
Extracting ./t10k-images-idx3-ubyte.gz
Extracting ./t10k-labels-idx1-ubyte.gz
Epoch: 001 | AvgCost: 0.476 | Train/Valid ACC: 0.903/0.909
Epoch: 002 | AvgCost: 0.339 | Train/Valid ACC: 0.911/0.918
Epoch: 003 | AvgCost: 0.320 | Train/Valid ACC: 0.915/0.922
Epoch: 004 | AvgCost: 0.309 | Train/Valid ACC: 0.918/0.923
Epoch: 005 | AvgCost: 0.301 | Train/Valid ACC: 0.918/0.922
Epoch: 006 | AvgCost: 0.296 | Train/Valid ACC: 0.919/0.922
Epoch: 007 | AvgCost: 0.291 | Train/Valid ACC: 0.921/0.925
Epoch: 008 | AvgCost: 0.287 | Train/Valid ACC: 0.922/0.925
Epoch: 009 | AvgCost: 0.286 | Train/Valid ACC: 0.922/0.926
Epoch: 010 | AvgCost: 0.283 | Train/Valid ACC: 0.923/0.926
Epoch: 011 | AvgCost: 0.282 | Train/Valid ACC: 0.923/0.924
Epoch: 012 | AvgCost: 0.278 | Train/Valid ACC: 0.925/0.927
Epoch: 013 | AvgCost: 0.278 | Train/Valid ACC: 0.925/0.928
Epoch: 014 | AvgCost: 0.276 | Train/Valid ACC: 0.925/0.925
Epoch: 015 | AvgCost: 0.276 | Train/Valid ACC: 0.926/0.928
Epoch: 016 | AvgCost: 0.274 | Train/Valid ACC: 0.927/0.927
Epoch: 017 | AvgCost: 0.270 | Train/Valid ACC: 0.927/0.925
Epoch: 018 | AvgCost: 0.273 | Train/Valid ACC: 0.927/0.930
Epoch: 019 | AvgCost: 0.270 | Train/Valid ACC: 0.927/0.929
Epoch: 020 | AvgCost: 0.268 | Train/Valid ACC: 0.927/0.927
Epoch: 021 | AvgCost: 0.268 | Train/Valid ACC: 0.927/0.926
Epoch: 022 | AvgCost: 0.270 | Train/Valid ACC: 0.928/0.926
Epoch: 023 | AvgCost: 0.268 | Train/Valid ACC: 0.927/0.926
Epoch: 024 | AvgCost: 0.266 | Train/Valid ACC: 0.929/0.926
Epoch: 025 | AvgCost: 0.261 | Train/Valid ACC: 0.927/0.926
Epoch: 026 | AvgCost: 0.269 | Train/Valid ACC: 0.929/0.927
Epoch: 027 | AvgCost: 0.265 | Train/Valid ACC: 0.928/0.928
Epoch: 028 | AvgCost: 0.261 | Train/Valid ACC: 0.929/0.928
Epoch: 029 | AvgCost: 0.266 | Train/Valid ACC: 0.930/0.926
Epoch: 030 | AvgCost: 0.261 | Train/Valid ACC: 0.929/0.924
Test ACC: 0.925