Preparing the Voxforge database

This notebook will demonstrate how to prepare the free Voxforge database for training. This database is a medium sized (~80 hours) database available online for free under the GPL license. A much more common database used in most research is the TIMIT, but that costs $250 and also much smaller (~4h - although much more professionally developed than Voxforge). The best alternative today is the Librispeech database, but that has a few dozen GB of data (almost 1000h) and wouldn't be sensible for a simple demo. So Voxforge it is...

First thing to do is realize what a speech corpus actually is: in its simplest form it is a collection of audio files (containing preferably speech only) with a set of transcripts of the speech. There are a few extensions to this that are worth noting:

• phonemes - transcripts are usually presented as a list of words - although not a rule, it is often easier to start the recognition process with phonemes and go from there. Voxforge defines a list of 39 phonemes (+ silence) and contains a lexicon mapping the words into phonemes (more about that below)
• aligned speech - the transcripts are usually just a sequence of words/phonemes, but they don't denote which word/phoneme occurs when - there are models that can learn from that (seq. learning, e.g. CTC, attention models), but having alignments is usually a big plus. TIMIT was hand-aligned by a group of professionals (which is why its a popular resource for research), but Voxforge wasn't. Fortunately, we can use one of the many available tools to do this automatically (with a margin of error - more on that below)
• meta-data - each recording session in the Voxforge database contains a readme file with useful information about the speaker and the environment that the recording took place in. When making a serious speech recognizer, this information can be very useful (e.g. for speaker adaptation - taking into account the speaker id, gender, age, etc...)

Downloading the corpus

To start working with the corpus, it needs to be downloaded first. All the files can be found in the download section of the Voxforge website under this URL:

http://www.repository.voxforge1.org/downloads/SpeechCorpus/Trunk/Audio/Main/16kHz_16bit/

There are 2 versions of the main corpus: sampled at 16kHz and 8kHz. The 16 kHz one is of better quality and is known as "desktop quality speech". While the original recordings were made at an even higher quality (44.1 kHz), 16k is completely sufficient for recoginzing speech (higher quality doesn't help much). 8 kHz is known as the telephony quality and is a standard value for the old (uncompressed, aka T0) digital telephone signal. If you are making a recognizer that has to work in the telephony environment, you should use this data instead

To download the whole dataset, a small program in Python is included in this demo. Be warned, this can take a long time (I think Voxforge is throttling the speed to save on costs) and restarts may be neccessary. The python method does check for failed downloads (compares file sizes) and restarts whatever wasn't downloaded completely, so you can run the method 2-3 times to make sure everything is ok.

Alternatively, wou can use a program like wget and enter this command (where "audio" is the dir to save the data to):

wget -P audio -l 1 -N -nd -c -e robots=off -A tgz -r -np http://www.repository.voxforge1.org/downloads/SpeechCorpus/Trunk/Audio/Main/16kHz_16bit

First lets import all the voxforge methods from the python directory. These will need the following libraries installed on your system:

• numpy - for working with data
• random, urllib, lxml, os, tarfile, gzip, re, pickle, shutil - these are standard system libraries and anyone should have them
• scikits.audiolab - to load the audio files from the database (WAV and FLAC files)
• tqdm - a simple library for progressbars that you can install using pip

Ignore any warnings above (I coudn't be bothered to compile audiolab with Alsa). Below you will find the method to download the Voxforge database. You only need to do this once, so you can run it either here or from a console or use wget. Be warned that it takes a long time (as mentioned earlier) so it's a good idea to leave it running over night.

Loading the corpus

Once the data is downloaded and stored in the 'audio' subdir of the main project dir, we can start loading the data into a Python datastructure. There are several methods that can be used for that. The following method will load a file and display its contents:

The loadBySpeaker method will load the whole folder and organize its contents by speakers (as a dictionary). Each utterance contains only the data and the prompts. For this demo, only 30 files are read - as this isn't a method we are going to ultimately use.

The corpus can also be extended by the phonetic transcription of the utterances using a lexicon file. Voxforge does provide such a file on its website and it is downloaded automatically (if it doesn't already exist).

Note that a single word can have several transcriptions. In the lexicon, these alternatives will have sequential number suffixes added to the word (word, word2, word3, etc), but this particular function will do nothing about that. Choosing the right pronounciation variant has to be done either manually, or by using a more sophisticated program (a pre-trained ASR system) to choose the right version automatically.

Aligned corpus

As mentioned earlier, this sort or cropus has it's downsides. For one, we don't know when each phoneme occurs so we cannot train the system discriminatavely. While it's still possible, it would be nice if we could start with a simpler example. Another problem is choosing the right pronounciation variant mentioned above.

To solve these issues, an automatic alignement was created using a different ASR system called Kaldi. This system is a very good ASR solution that implements various types of models. It also contains simple out-of-the-box scripts for training on Voxforge data.

To create the alignments using Kaldi, a working system had to be trained first and what's interesting, the same Voxforge data was used to train the system. How was this done? Well, Kaldi uses (among other things) a classic Gaussian Mixture Model and trains it using the EM algorithm. Initially the alignment is assumed to be even, throughout the file, but as the system is trained iteratively, the model gets better and thus the alignment gets more accurate. The system is trained with gradually better models to achieve even more accurate results and the provided solution here is generated using the "tri3b" model, as described in the scripts.

The alignments in Kaldi are stored in special binary files, but there are simple tools to help convert them into something more easier to use. The type of file chosen for this example is the CTM file, which contains a series of lines in a text file, each line describing a single word or phoneme. The description has 5 columns: encoded file name, unused id (always 1), segment start, segment length and segment text (i.e. word of phoneme name/value). This file was generated using Kaldi, compressed using gzip and stored in 'ali.ctm.gz' in the 'data' directory of this project.

Please note, that the number of files in this aligned set is smaller than the acutal count in the whole Voxforge dataset. This is because there is a small percentage of errors in the database (around a 100 files or so) and some recordings are of such poor quality that Kaldi couldn't generate a reasonable alignemnet for these files. We can simply ignore them here. This, however, doesn't mean that all the alignments present in the CTM are 100% accurate. There can still be mistakes there, but hopefully they are unlikely enough to not cause any issue.

While this file contains everything that we need, it'd be useful to convert it into a datastructure that can be easily used in Python. The convertCTMToAli method is used for that:

We store the generated datastructure into a gzipped and pickled file, so we don't need to perform this more than once. This file is already included in the repository, so you can skip the step above.

We can read the file like this:

Here is an example of the structure and its attributes loaded from that file:

Please note that the audio data is not yet loaded at this step (it's set to None).

Test data

Before we go on, we need to prepare our test set. This needs to be completely independent from the training data and it needs to be the same for all the experiemnts we want to do, if we want to be able to make them comparable in any way. The test set also needs to be "representable" of the whole data we are working on (so they need to be chosen randomly from all the data).

This isn't the only way we could perform our experiments - very often people use what is known as "k-fold cross validation", but that would take a lot of time to do for all our experiemnts, so choosing a single representative evaluation set is a more convinient option.

Now, generally most corpora have a designated evaluation set: for example, TIMIT has just such a set of 192 files that is used by most papers on the subject. Voxforge doesn't have anything like that and there aren't many papers out there using it as a resource anyway. One of the most advanced uses of Voxforge is in Kaldi and there they only shuffle the training set and choose 20-30 random speakers from that. To make our experiemtns at least "comparable" to TIMIT, we will try and do a similar thing here, but we will save the list of speakers (and their files) so anyone can use the same when conducting their experiemnts.

WARNING If you want to compare the results of your own experiemnts to the ones from these notebooks, then don't run the code below and use the files provided in the repo. If you run the code below, you will reset the test ordering and your experiements won't be strictly comparable to the ones from these notebooks.

To make things more managable for this demo, we will take 5% of the training set and work using that instead of the whole 80 hours. 5% should give us an amount similar to TIMIT. If you wish to re-run the experiments using the whole dataset, go to the bottom of this notebook for further instructions.

Here we load additional data using the loadAlignedCorpus method. It loads the alignment and the appropriate audio datafile for each utterance (it can take a while for larger corpora):

We have to do the same for the test data:

Now we can check if we have all the neccessary data: phonemes, phoneme alignments and data.

Feature extraction

To perform a simple test, we will use a standard set of audio features used in many, if not most papers on speech recognition. This set of features will first split each file into a bunch of small chunks of equal size, giving about a 100 of such frames per second. Each chunk will then be converted into a vector of 39 real values. Furhtermore, each vector will be assigned a phonetic class (value from 0..39) thanks to the alignemnt created above. The problem can then be sovled as a simple classification problem that maps a real vector to a phonetic class.

This particular set of features is calculated to match a specification developed in a classic toolkit known as HTK. All the details on this feature set can be found under the linked repository here. If you want to experiment with this feature set (highly encouraged) please read the description there.

In the code below, we extract the set of features for each utterance and store the results, together with the classification decision for each frame. For performance reasons, we will store all the files in HDF5 format using the h5py library. This will allow us to read data directly from the drive without wasting too much RAM. This isn't as important when doing small experiemnts, but it will get relevant for doing the large ones.

The structure of the HDF5 file is broken into utterances. The file contains a list of utterances sotred as groups in the root and each utterance has 2 datasets: inputs and outputs. Later also normalized inputs are added.

Since we intend to use this procedure more than once, we will encapsulate it into a function:

Now let's process the small training and test datasets:

Normalization

While usable as-is, many machine learning models will perform badly if the data isn't standardized. Standarization or normalization stands for making sure that all the samples are distributed to a reasonable scale - usually centered around 0 (with a mean of 0) and spread to a standard deviation of 1. The reason for this is because the data can come from various sources - some people are louder, some are quiter, some have higher pitched voices, some lower, some used a more sensitive microphones than other, etc. That is why the audio between sessions can have various ranges of values. Normalization makes sure that all the recordings are tuned to a similar scale before processing.

To perform normalization we simply need to compute the mean and standard deviation of the given signal and then subtract the mean and divide by the standard deviation (thus making the new mean 0 and new stdev 1). A common question is what signal do we use to perform these calculations? Do we calculate it once for the whole corpus, or once per each utterance? Maybe once per speaker or once per session? Or maybe several times per utterance?

Generally, the longer the signal we do this on, the better (the statistics get more accurate), but performing it only once on the whole corpus doesn't make much sense because of what is written above. The reason we normalize the data is to remove the differences between recording sessions, so at minimum we should normalize each session seperately. In practice, it's easier to just normalize each utterance as they are long enough on their own. This is known as "batch normalization" (where each utterance is one batch).

But this makes one assumption, that the recording conditions don't change significantly throughout the whole utterance. In certain cases, it may actually be a good idea to split the utterance into several parts and normalize them sepeartely, in case the volume changes throught the recording, or maybe there is more than one speaker in a single file. This is solved best by using a technique known as "online normalization" which uses a sliding window to compute the statistics and can react to rapid changes in the values. This is, however, beyond the scope of this simple demo (and shouldn't really be neccessary for this corpus anyway).

To see what's inside we can run the following command in the terminal:

Simple classification example

To finish here, we will use a simple SGD classifier from the scikit.learn library to classify the phonemes from the database. We have all the datasets prepared above, so all we need to do is load the preapared arrays. We will use a special class called Corpus included in the data.py file. In the constructor we provide the path to the file and say that we wish to load the normalized inputs. Next we use the get() method to load the list of all the input and out values. This method returns a tuple - one value for inputs and one for outputs. Each of these is a list of arrays corresponding to individual utterances. We can then convert it into a single contiguous array using the concatenate and vstack methods:

Here we create the SGD classifier model. Please note that the settings below work on the version 0.17 of scikit-learn, so it's recommended to upgrade. If you can't, then feel free to modify the settings to something that works for you. You may also turn on verbose to get more information on the training process. Here it's off to preserve space in the notebook.

Here we train the model. It took 4 minutes for me:

Here we get about ~52% accuracy which is pretty bad for phoneme recogntion. In other notebooks, we will try to improve on that.

Other data

Here we will also prepare the rest of the data to perform other experiments. If you wish to make only simple experiments and don't want to waste time on preparing large datasets and wasting a lot of time, feel free to skip these steps. Be warned that the dataset for the full 80 hours of training data takes up to 10GB in so you will need that much memory in RAM as well as your drive to make it work using the code present in this notebook.