Our challenge is to predict which button a user is going to click based on previous mouse movements. One application might be to highlight the button before hover and give a clearer guidance to the user:
Even though the model will vary between different machines and different users, we have a default data set you can use to train the model.
Read the complete article here https://dev.to/djcordhose/improving-user-experience-with-tensorflow-js-4693 and have a look at the browser application http://djcordhose.github.io/ux-by-tfjs/dist that can load our model to make online predictions.
In [1]:
# for colab
!pip install -q tf-nightly-gpu-2.0-preview
In [3]:
import tensorflow as tf
print(tf.__version__)
In [4]:
# a small sanity check, does tf seem to work ok?
hello = tf.constant('Hello TF!')
print("This works: {}".format(hello))
In [5]:
# this should return True even on Colab
tf.test.is_gpu_available()
Out[5]:
In [6]:
tf.test.is_built_with_cuda()
Out[6]:
In [7]:
tf.executing_eagerly()
Out[7]:
In [8]:
import pandas as pd
print(pd.__version__)
In [0]:
# local
# URL = '../data/sample4.json'
# remote
URL = 'https://raw.githubusercontent.com/DJCordhose/ux-by-tfjs/master//data/sample4.json'
df = pd.read_json(URL, typ='series')
In [10]:
len(df)
Out[10]:
In [11]:
df.head()
Out[11]:
In [0]:
X = [item['x'] for item in df]
In [13]:
X[0]
Out[13]:
In [0]:
y = [item['y'] - 1 for item in df]
In [15]:
y[0]
Out[15]:
In [0]:
from math import floor
def make_chunks(list_to_chunk, chunk_size):
length = len(list_to_chunk)
assert length / chunk_size == floor(length / chunk_size), "length of data must be multiple of segment length"
for chunk_start in range(0, length, chunk_size):
yield list_to_chunk[chunk_start : chunk_start + chunk_size]
In [0]:
import numpy as np
CHUNK_SIZE = 25
# only use the final segments
SEGMENTS = 2
X_expanded = []
y_expanded = []
for x_el, y_el in zip(X, y):
chunks = list(make_chunks(x_el, CHUNK_SIZE))
chunks = chunks[len(chunks) - SEGMENTS:]
labels = [y_el] * SEGMENTS
for seq, label in zip(chunks, labels):
X_expanded.append(seq)
y_expanded.append(label)
X_expanded = np.array(X_expanded)
y_expanded = np.array(y_expanded)
In [18]:
X_expanded.shape
Out[18]:
In [19]:
X_expanded[100]
Out[19]:
In [20]:
X_expanded[100][0]
Out[20]:
In [21]:
y_expanded[100]
Out[21]:
In [22]:
np.unique(y_expanded)
Out[22]:
In [0]:
assert np.array_equal(np.unique(y_expanded), [0, 1, 2])
RNNs work on sequences of input data and can learn which part of the history is relevant. Thus, they can learn which mouse events are relevant.
Inexpensive in terms of model size and compute, but are bad at remembering events far in the past.
Can learn which events in the past are important and can thus much better match a certain mouse path. However, 4x more expensive than the simple RNN.
Like LSTMs, but only 3x more expensive. Often as good as LSTMs.
In [0]:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import Dense, LSTM, GRU, SimpleRNN, BatchNormalization
from tensorflow.keras.models import Sequential, Model
In [0]:
# experiment with
# - type of RNN: SimpleRNN, LSTM, GRU
# - number of units
# - dropout
# - BatchNormalization: yes/no
n_steps = len(X_expanded[0])
n_features = len(X_expanded[0][0])
n_buttons = 3
model = Sequential()
model.add(SimpleRNN(units=50, activation='tanh', input_shape=(n_steps, n_features), name="RNN_Input",
# model.add(GRU(units=50, activation='tanh', input_shape=(n_steps, n_features), name="RNN_Input",
# recurrent_dropout makes things slow
# dropout=0.1, recurrent_dropout=0.1))
dropout=0.1))
# model.add(GRU(units=50, activation='tanh', input_shape=(n_steps, n_features), name="RNN_Input"))
model.add(BatchNormalization())
model.add(Dense(units=n_buttons, name='softmax', activation='softmax'))
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
In [26]:
%%time
EPOCHS = 200
BATCH_SIZE = 200
history = model.fit(X_expanded, y_expanded,
batch_size=BATCH_SIZE,
epochs=EPOCHS, verbose=0, validation_split=0.2)
In [27]:
loss, accuracy = model.evaluate([X_expanded], y_expanded, batch_size=BATCH_SIZE)
accuracy
Out[27]:
In [28]:
%matplotlib inline
import matplotlib.pyplot as plt
# plt.yscale('log')
plt.ylabel('loss')
plt.xlabel('epochs')
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.legend(['loss', 'val_loss'])
Out[28]:
In [29]:
plt.ylabel('accuracy')
plt.xlabel('epochs')
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.legend(['accuracy', 'val_accuracy'])
Out[29]:
In [30]:
model.predict([[X_expanded[0]]])
Out[30]:
In [31]:
model.predict([[X_expanded[0]]]).argmax()
Out[31]:
In [32]:
y_expanded[0]
Out[32]:
In [0]:
y_pred = model.predict([X_expanded]).argmax(axis=1)
In [34]:
cm = tf.math.confusion_matrix(labels=y_expanded, predictions=y_pred)
cm
Out[34]:
In [35]:
import seaborn as sns
classes = ["Left Button", "Middle Button", "Right Button"]
sns.heatmap(cm, annot=True, fmt="d", xticklabels=classes, yticklabels=classes)
Out[35]:
In [0]:
model.save('ux.hd5')
In [37]:
!ls -l
In [38]:
!pip install -q tensorflowjs
In [0]:
!tensorflowjs_converter --input_format keras ux.hd5 tfjs
In [40]:
!ls -l tfjs
Download using Files menu on the left