notebook.community

Edit and run

Time-Delay Neural Network

The Time-Delay Neural Network (TDNN) is a model that is able to perform sequence recognition. It maps the buffered sequence into a spatial pattern on the input layer of a Multilayer Perceptron (MLP). In essence, it is a convolutional neural network. Despite the natural association of a sequence with time, any other dynamic variable can be used, such as the DNA, a grammatical structure, etc.

There are two main drawbacks to this general approach to sequence recognition:

  • It is unable to deal with arbitrary sequence lengths. The size of the delay line or shift register must be set in advance.
  • Every additional shift (thus, input neuron) adds H parameters to the network (more expressiveness), which increases the size of the training set for successful learning and generalisation.

Time-dependent signals

In the case of time-dependent signals, the restrictions defined by the sampling theorem apply to them. The signals must be adequately sampled, otherwise, a poor representation (including distortions) may have a negative impact on the learning capacity of the TDNN. However, there exists a way to compensate for this problem and make the system more robust. The constant delays (determined by the sampling period) can be replaced by spreading filters $g(\cdot)$ that broaden the signal in time as well as delaying it. This filter is defined as follows:

$$S = \sum_{i=0}^{N} \left ( \frac{i}{d+1} \right ) ^ \alpha \exp \left ( \alpha \frac{1 - i}{d + 1} \right )$$

where $N$ is the size of the delay line (buffer), $\alpha$ is the spreading factor, and $d$ is the shift delay number. Note that $S$ is a normalisation factor that ensures that all shifts may deliver the sample amount of energy. Then, for each shift:

$$g(x[n-d]) = g(d) = \frac{1}{S} \sum_{i=0}^{N} x[i] \left ( \frac{i}{d+1} \right ) ^ \alpha \exp \left ( \alpha \frac{1 - i}{d + 1} \right )$$

The application of the spreading filter can be seen as a fixed weight matrix on the signal buffer. The spreading factor $\alpha$ defines the sharpness of the filter (positive relation). What follows is a graphical representation of each filter shape.





In [9]:



    


import NeuralNetwork

%matplotlib inline
import matplotlib.pyplot as plt

# 10 is alpha filter spreading factor, 8 sequence length (input neurons), 3 hidden neurons, 2 output neurons
tdnn = NeuralNetwork.TDNN(20, [8,3,2])

plt.figure()
colors = ['sienna', 'red', 'orange', 'gold', 'chartreuse', 'turquoise', 'blue', 'purple']
g = tdnn[0]
plt.subplot(1,2,1)
for i in xrange(g.shape[0]):
    plt.plot(g[i], colors[i], label='G(' + str(i) + ')', linewidth=2)
ax = plt.gca()
ax.set_ylim([0,1.0])
plt.legend(loc='upper right')
plt.xlabel('Sequence [n]')
plt.ylabel('Gain')
plt.title('G(n) with alpha=20')

# broader filter, alpha = 4
tdnn = NeuralNetwork.TDNN(4, [8,3,2])

colors = ['sienna', 'red', 'orange', 'gold', 'chartreuse', 'turquoise', 'blue', 'purple']
g = tdnn[0]
plt.subplot(1,2,2)
for i in xrange(g.shape[0]):
    plt.plot(g[i], colors[i], label='G(' + str(i) + ')', linewidth=2)
ax = plt.gca()
ax.set_ylim([0,1.0])
plt.legend(loc='upper right')
plt.xlabel('Sequence [n]')
plt.title('G(n) with alpha=4')

plt.show()

In order to prove the effectiveness of this approach, the following experiment is going to sample a signal of 1Hz (class 1) and another one of 4Hz (class 2) at 10Hz, and then it will see how the TDNN works with or without the spreading filter, and with a sampling frequency jitter of 2Hz (so, the sampling rate can be any value from 8Hz to 12Hz). Note that the 4Hz signal can match Nyquist's frequency if the sampling frequency is 8Hz.





In [10]:



    


import numpy as np

# No sampling jitter, no spreading filter

nsamp = 50
train_dat = []
train_lab = []
test_dat = []
test_lab = []

for i in xrange(nsamp):
    train_dat.append(np.sin(2*np.pi*1.0/10.0 * np.array(range(8)) + np.random.rand()*2*np.pi))
    train_lab.append([0.79, 0.21])
    train_dat.append(np.sin(2*np.pi*4.0/10.0 * np.array(range(8)) + np.random.rand()*2*np.pi))
    train_lab.append([0.21, 0.79])

train_dat = np.array(train_dat)
train_lab = np.array(train_lab)

mlp = NeuralNetwork.MLP([8,4,2])
NeuralNetwork.MLP_Backprop(mlp, train_dat, train_lab, 0.1, 20, 0.1)

print("Test")
accuracy = 0
for i in xrange(100):
    res = NeuralNetwork.MLP_Predict(mlp, np.sin(2*np.pi*1.0/10.0 * np.array(range(8)) + np.random.rand()*2*np.pi))[2]
    if res[0] > res[1]:
        accuracy = accuracy + 1
    res = NeuralNetwork.MLP_Predict(mlp, np.sin(2*np.pi*4.0/10.0 * np.array(range(8)) + np.random.rand()*2*np.pi))[2]
    if res[1] > res[0]:
        accuracy = accuracy + 1
print("Accuracy = " + str(float(accuracy)/2) + "%")



    


















    






J(0) = 0.170594648820766
J(1) = 0.1640330153378083
J(2) = 0.16127056862028138
J(3) = 0.1587691789275572
J(4) = 0.15627792420719835
J(5) = 0.15374988048888177
J(6) = 0.15116421296959837
J(7) = 0.1485080265684253
J(8) = 0.14577055208508316
J(9) = 0.14294049230703487
J(10) = 0.1400053049912802
J(11) = 0.13695192053591143
J(12) = 0.13376853289306484
J(13) = 0.13044704715565061
J(14) = 0.12698557678797628
J(15) = 0.12339023616514043
J(16) = 0.1196756021522355
J(17) = 0.11586367144165832
J(18) = 0.1119816711865667
J(19) = 0.10805936956808386
Elapsed time = 0.597797870636 seconds
Test
Accuracy = 81.0%

















In [11]:



    


# Sampling jitter, no spreading filter

train_dat = []
train_lab = []
test_dat = []
test_lab = []

for i in xrange(nsamp):
    train_dat.append(np.sin(2*np.pi*1.0/(8.0 + np.random.rand()*4) * np.array(range(8)) + np.random.rand()*2*np.pi))
    train_lab.append([0.79, 0.21])
    train_dat.append(np.sin(2*np.pi*4.0/(8.0 + np.random.rand()*4) * np.array(range(8)) + np.random.rand()*2*np.pi))
    train_lab.append([0.21, 0.79])

train_dat = np.array(train_dat)
train_lab = np.array(train_lab)

mlp = NeuralNetwork.MLP([8,4,2])
NeuralNetwork.MLP_Backprop(mlp, train_dat, train_lab, 0.1, 20, 0.1)

print("Test")
accuracy = 0
for i in xrange(100):
    res = NeuralNetwork.MLP_Predict(mlp, np.sin(2*np.pi*1.0/(8.0 + np.random.rand()*4) * np.array(range(8)) + np.random.rand()*2*np.pi))[2]
    if res[0] > res[1]:
        accuracy = accuracy + 1
    res = NeuralNetwork.MLP_Predict(mlp, np.sin(2*np.pi*4.0/(8.0 + np.random.rand()*4) * np.array(range(8)) + np.random.rand()*2*np.pi))[2]
    if res[1] > res[0]:
        accuracy = accuracy + 1
print("Accuracy = " + str(float(accuracy)/2) + "%")



    


















    






J(0) = 0.1724422964588266
J(1) = 0.16264486918921314
J(2) = 0.15759184461908152
J(3) = 0.1535569556721732
J(4) = 0.1499841702799275
J(5) = 0.14655005515597697
J(6) = 0.14301673075183133
J(7) = 0.13923746859608882
J(8) = 0.13516328246430706
J(9) = 0.13083389751319485
J(10) = 0.12635007340055118
J(11) = 0.12183731523959214
J(12) = 0.11741500966164722
J(13) = 0.11317829785829339
J(14) = 0.10919191548380727
J(15) = 0.10549174616945078
J(16) = 0.10209004004731122
J(17) = 0.09898159652179837
J(18) = 0.09614943830953529
J(19) = 0.09356934923668636
Elapsed time = 0.559809923172 seconds
Test
Accuracy = 45.0%

















In [12]:



    


# Sampling jitter, spreading filter

train_dat = []
train_lab = []
test_dat = []
test_lab = []

for i in xrange(nsamp):
    train_dat.append(np.sin(2*np.pi*1.0/(8.0 + np.random.rand()*4) * np.array(range(8)) + np.random.rand()*2*np.pi))
    train_lab.append([0.79, 0.21])
    train_dat.append(np.sin(2*np.pi*4.0/(8.0 + np.random.rand()*4) * np.array(range(8)) + np.random.rand()*2*np.pi))
    train_lab.append([0.21, 0.79])

train_dat = np.array(train_dat)
train_lab = np.array(train_lab)

tdnn = NeuralNetwork.TDNN(4, [8,4,2])
NeuralNetwork.TDNN_Backprop(tdnn, train_dat, train_lab, 0.1, 20, 0.1)

print("Test")
accuracy = 0
for i in xrange(100):
    res = NeuralNetwork.TDNN_Predict(tdnn, np.sin(2*np.pi*1.0/(8.0 + np.random.rand()*4) * np.array(range(8)) + np.random.rand()*2*np.pi))[2]
    if res[0] > res[1]:
        accuracy = accuracy + 1
    res = NeuralNetwork.TDNN_Predict(tdnn, np.sin(2*np.pi*4.0/(8.0 + np.random.rand()*4) * np.array(range(8)) + np.random.rand()*2*np.pi))[2]
    if res[1] > res[0]:
        accuracy = accuracy + 1
print("Accuracy = " + str(float(accuracy)/2) + "%")



    


















    






J(0) = 0.18218090091226785
J(1) = 0.17126904940877805
J(2) = 0.1700864251432075
J(3) = 0.16976099439599868
J(4) = 0.16955344178829637
J(5) = 0.1693841076747113
J(6) = 0.1692369897248332
J(7) = 0.1691059424443806
J(8) = 0.16898747649172752
J(9) = 0.16887920073840593
J(10) = 0.16877932751950492
J(11) = 0.16868646006955104
J(12) = 0.16859947656778157
J(13) = 0.16851745636114063
J(14) = 0.16843962898166084
J(15) = 0.168365337352417
J(16) = 0.1682940105042485
J(17) = 0.16822514288887713
J(18) = 0.16815827833167138
J(19) = 0.1680929972566302
Elapsed time = 0.541599035263 seconds
Test
Accuracy = 72.0%

For a given set of conditions, the TDNN with the spreading filter proves to be robust and resilient to signal issues.

Content source: ai-maker/ntk

Similar notebooks:

notebook.community | gallery | about