Supervised Classification of Audio Streams: Part I

Topic goal

We will explore how to classify audio streams to make a touch sensor from a microphone, using supervised machine learning approaches. This introduces classification as a way of building controls from sensors, how to evaluate performance meaningfully, and the issues that are encountered in turning time series like audio into usable inputs.

Outline

In the next two hours, we will:

[Part I]

• Discuss acoustic touch sensing
• Quickly review machine learning, and supervised classification
• Examine how to evaluate a classifier
• Discuss feature transforms for audio

• **Practical**: build a simple binary classifier to discriminate two scratching sounds.

[Part II]

• Discuss overfitting and how to avoid it
• See how to split test and training data for time series.
• Discuss ensembling techniques
• Look at more advanced audio features

• **Challenge**: build the **best** acoustic touch classifier for Stane-like **acoustic touch sensing** waveforms.

Motivation

This topic is essentially the operations behind the Stane project Paper Video

This used 3D printed textures on mobile devices. Scratching the fingernail across the surface generates distinctive sounds, which are propagated through the case and picked up by a piezo microphone.

Different regions have different textures, and thus the area being rubbed can be determined by analysing the audio signal.

What is machine learning?

Machine learning can be summarised as making predictions from data

Machine learning focuses on trying to estimate the value of unknown variables given some data which might predict them.

Machine learning algorithms generally have a training phase, where input data is used to update the parameters of a function that tries to model the data, and a testing phase, where data unseen by the learning algorithm is used to evaluate how well the model is able to do prediction.

Supervised learning

Supervised learning involves learning a relationship between attribute variables and target variables; in other words learning a function which maps input measurements to target values. This can be in the context of making discrete decisions (is this image a car or not?) or learning continuous relationships (how loud will this aircraft wing be if I make the shape like this?).

Some mathematical notation

We consider datasets which consist of a series of measurements. We learn from a training set of data. Each measurement is called a sample or datapoint, and each measurement type is called a feature.

If we have $n$ samples and $d$ features, we form a matrix $X$ of size $n \times d$, which has $n$ rows of $d$ measurements. $d$ is the dimension of the measurements. $n$ is the sample size. Each row of $X$ is called a feature vector. For example, we might have 200 images of digits, each of which is a sequence of $8\times8=64$ measurements of brightness, giving us a $200 \times 64$ dataset. The rows of image values are the features. In a supervised learning situation, we will also have a vector of targets $Y$. These will be the output values we assign to each of the training feature vectors; one target per row of the training features.

We want to learn a function $$y^\prime = f(x^\prime)$$ which works for a value $x^\prime$ that we have not seen before; i.e. we want to be able to predict a value $y^\prime$ based on a model ($f(x)$) that we learned from data. If we are doing classification, these targets will be categorical labels e.g. [0,1,2,3].

Simple learning -- the perceptron

Let's look at a quick example of a simple supervised learning task -- binary classification. That is we take some training features $X$ and some binary indicator labels $Y$ and learn a function $f(x)$ which has two possible outputs: [0,1] (or equivalently [-1,1]).

A very simple such classifier is the perceptron which attempts to find a linear weighting of the inputs such that the sign of the output matches the class label. In 2D this would be drawing a line between the two classes; in the 3D a plane and so on. The function that will be learned is linear (i.e. is just a weighting of the inputs). Since there is only one output variable, the weights a 1D vector, and these weights are denoted $w$. There are $d$ weights, one for each feature. The function to be learned is of the form: $$f(x) = \text{sgn}(w^Tx),$$ where $\text{sgn}$ is the sign function and $w^T$ is the weight vector transposed so as to form the weighted sum.

We can get some insight into what the perceptron is doing by plotting the decision boundary in the feature space. This shows us which parts of the space the classifier indicates are +1 and which are -1. We do this just by evaluating $f(x)$ over a grid of points.

Limitations

The perceptron is very simple, but can only learn functions that divide the feature space with a hyperplane. If the datapoints to be classified have classes that cannot be separated this way in the feature space, the perceptron cannot learn the function.

Evaluating without deceiving yourself

We need to be able to quantify how well our learning algorithms perform on predicting unseen data given the model that has been learned. This involves testing on data that was not presented to the learning algorithm during the training phase.

This means you must ALWAYS split your data in completely separate training and test sets. Train on the training data to get a model which you test on the test set. NEVER test on data you trained on -- we'll discuss this more after the practical.

Classifiers

An obvious metric is accuracy, the ratio of correctly classified examples to the total number of examples.

Why is accuracy not enough?

This is an easy to interpret but sometimes insufficient metric for performance. One common situation where it fails is where the dataset is not balanced (e.g. there are many more examples for one label than another). If 95% of the dataset are of class 0, and 5% of class 1, predicting 0 regardless of the input has a 95% accuracy rate.

Receiver-operator curves

A very useful tool for capturing binary classification performance is the receiver-operator curve (ROC curve). This works with any classifier that produces scores as outputs (e.g. continuous values in the range [0,1]). Classifiers that only produce discrete class labels cannot be used to generate a ROC curve.

To plot the curve, we iterate through a set of threshold values $\tau_1, \tau_2, \dots$, and plot the accuracy we would get if we thresholded the classifiers at $\tau_i$. A classifier with chance performance will have an ROC curve with $y=x$; a very good classifier will have the curve bent up towards the upper-left corner.

AUC

The area under the curve (AUC) of the ROC curve (i.e. the integral of the ROC curve) is a useful summary metric for performance. An AUC of 1.0 indicates perfect classification. An AUC of 0.5 indicates chance performance.

Confusion matrices for multiclass problems

Confusion matrices are effective tools for communicating where classifiers are going wrong for multi-class problems: i.e. which labels are being confused with which? A confusion matrix shows the distribution of predicted labels for each true label as a matrix of size $k \times k$ for $k$ labels.

Perfect classification results in a confusion matrix with a single diagonal of 1s (every test example predicts the label to be the true label). This matrix can reveal classes which are poorly separated in multi-class problems.

Processing time series for classification

Feature vectors

In almost all machine learning contexts, we predict outputs given a fixed length set of features; the input space a fixed dimension $d$. Each of those features is usually (but not always) continuous-valued.

Sometimes the data fall naturally into this space (e.g. classifying the iris type by 3 physical measurements). In cases such as in audio classification, though, we want to make predictions based on time series; a set of measurements of the same variable set made repeatedly over time.

Windowing

One general solution to this time series problem is to use a delay embedding -- a fixed length sequence of previous measurements. For example the measurements $[x_{t=t}, x_{t=t-1}, x_{t=t-2}, \dots, x_{t=t-d}]$ might make up the feature vector. If we just use $d$ consecutive measurements, this process is known as windowing, because we chop up the data into fixed length windows by "sliding" a time window along the data. Consecutive (but possible discontiguous or overlapping) windows are almost universally used in audio contexts.

For example, we might split a speech stream, recorded at 8Khz, into 160 sample (40ms) windows, and then try and classify each window as to what phoneme that window contains. The idea is that 20ms is enough to distinguish an "aah" sound from a "ssh" sound.

These windows can overlap, which increases the size of the training set, but excessive overlapping can capture lots of redundant features examples. This can increase overfitting and training time without improving the classifier performance. Balancing the size of the windows (and thus the feature vector size $d$) and the amount of overlap is a matter of experimentation and domain knowledge.

Feature transforms

Often the "natural" raw form of the data can be difficult to classify. This might be because it has very high dimension, it is very noisy, or the classification boundary just isn't very compatible with your classifier (e.g. the class borders in the original space are highly-nonlinear and you are using a linear classifier).

Feature engineering is the art of finding transforms of the raw data that increase classifier performance. These can often be simple, such as dropping some of the measurements entirely (under the assumption that they are irrelvant), or averaging measurements together (under the assumption that this reduces noise).

Audio transforms

Audio data tends to be very high dimensional -- you might get 4000 to 44100 measurements for a single second of data. A single audio sample has very little information indeed; it is the longer-term (millisecond to second) properties that have all the interesting information. In an HCI task we probably wouldn't expect there to be hundreds of interaction "events" or state changes in a second; humans can't control things that quickly.

So want transforms that pull out interesting features over time. The classical feature transform is the Fourier transform, which rewrites a signal varying over time as a sum of sinusoidal (periodic) components. This functions much like our ear works, splitting up audio in frequency bands, each of which has a phase and an amplitude.

Windowing/spectral leakage

The Fourier transform and related transforms expect an infinite, periodic signal. In audio classification, we have a short sample of a non-periodic signal (although there may well be periodic components). If we just take the FT of a chunk of a signal, there will be instaneous "jumps" at the start and end which corrupt the spectrum (this is called spectral leakage). Any transform which expects periodic or infinite data usually benefits from a window function.

Applying a window function to taper off the function eliminates this problem.

Practical exercise: a baseline classifier

Task: Build a simple audio classifier

You have two audio files: data/rub_1.wav and data/rub_2.wav which are acoustic microphone recordings from a Stane device being rubbed. The wave files are 8Khz, 16 bit, mono, and are 10 seconds long each.

The task is to build and evaluate a binary classifier that distinguishes the two sounds reliably. You should try this once with the raw audio features (i.e. no transform), and then see if you can get better performance using the FFT or other feature transforms.

Use one of these classifiers from sklearn (see the documentation here for details: http://scikit-learn.org/stable/supervised_learning.html#supervised-learning):

• KNeighborsClassifier
• SVC
• LinearDiscriminantAnalysis
• RandomForestClassifier

If you have no strong preference, try the LinearDiscriminatAnalysis classifier, which has no parameters to adjust.

Steps:

1. Get the files loaded and normalised (see below)
2. Set up the windowing of the data (see below for windowing function)
3. Set up a basic classifier
4. Get a ROC curve, AUC and accuracy up
5. Add an FFT magnitude transform
6. Retrain and test
7. Adjust windowing to improve performance

Tips:

• Load wavefiles using scipy.io.wavfile.read() (see below).
• A function window_data() is provided to window data to make fixed length features. You need to set the parameters! You should probably use a small value for subsample (e.g. 0.1) to make the process go faster, at least to begin with.
• You should compute accuracy, AUC and plot a ROC curve (see above for the code to do that).
• After trying out the raw data, you should try transforming the data. The np.fft module provides FFT functionality. Be aware that you need to feed real (as in, not complex!) data to the classifier -- if you take a complex FFT, you need to convert to a magnitude or phase representation (e.g. using np.abs(fft(window))) to get the magnitude spectrum or np.angle(fft(window) to get the phase spectrum)

Link to Audio Classification Part II