In [14]:
%matplotlib inline
from __future__ import division, print_function
import numpy as np
import matplotlib.pyplot as plt
import SHS_data
import evaluation
import main
import util
import fingerprints as fp
In [28]:
# %run main.py
In [6]:
cliques_by_name, cliques_by_uri = SHS_data.read_cliques()
In [15]:
# ratio = (1, 10, 90)
ratio = (5, 15, 80)
# ratio = (10, 25, 65)
train_cliques, test_cliques, val_cliques = util.split_train_test_validation(cliques_by_name, ratio=ratio)
In [39]:
reload(main)
reload(fp)
fp_function = fp.cov
results = main.run_leave_one_out_experiment(train_cliques, fp_function, print_every=50)
print('ratio:', ratio)
print('fp_function:', fp_function.__name__)
print('results:', results)
All dist_metric='cosine' unless stated otherwise.
results: {'mean r5': 0.055701801319778844, 'mean ap': 0.08220243505342123, 'mean p1': 0.10898876404494381}
results: {'mean r5': 0.071334586137956915, 'mean ap': 0.1027200457310438, 'mean p1': 0.11573033707865168}
Chroma + Chroma Delta covariance fingerprints
results: {'mean r5': 0.10796936521655622, 'mean ap': 0.093784444078998788, 'mean p1': 0.11573033707865168}
results: {'mean r5': 0.13988674870697343, 'mean ap': 0.12034266027019991, 'mean p1': 0.1303370786516854}
This should make no difference when distance metrix is 'cosine'.
results: {'mean r5': 0.13988674870697343, 'mean ap': 0.12034266027019991, 'mean p1': 0.1303370786516854}
Small difference of 0.002.
results: {'mean r5': 0.13393323592761797, 'mean ap': 0.12203557885856681, 'mean p1': 0.13820224719101123}
Slower! No big difference
results: {'mean r5': 0.13393323592761797, 'mean ap': 0.12203557885856681, 'mean p1': 0.13820224719101123}results: {'mean r5': 0.0019689112385741598,
'mean ap': 0.0096498583481854708,
'mean p1': 0.0022471910112359553}
Fourier with n_ftm, hop_length = 32, 16, mean-pooling:
results: {'mean r5': 0.11329444587871552,
'mean ap': 0.16503478256906676, **
'mean p1': 0.18089887640449437}
Fourier with n_ftm, hop_length = 16, 8, mean-pooling:
results: {'mean r5': 0.10124912540642877,
'mean ap': 0.14812587369978256,
'mean p1': 0.16292134831460675}
Fourier with n_ftm, hop_length = 64, 32, mean-pooling:
results: {'mean r5': 0.093912922283708816,
'mean ap': 0.13756700301430866,
'mean p1': 0.14157303370786517}
As in paper: n_ftm, hop_length = 75, 75, chroma squared, no log, median-pooling:
results: {'mean r5': 0.099972854860495314,
'mean ap': 0.15284109052120123,
'mean p1': 0.17078651685393259}
Sort of as in paper: n_ftm, hop_length = 75, 75, chroma squared, no log, MEAN-pooling:
results: {'mean r5': 0.11252996691198938,
'mean ap': 0.16674428398443461, ***
'mean p1': 0.18651685393258427}
Sort of as in paper: n_ftm, hop_length = 75, 75, chroma squared, log, MEAN-pooling:
results: {'mean r5': 0.065730418146148495,
'mean ap': 0.097336923877268319,
'mean p1': 0.085393258426966295}
Sort of as in paper: n_ftm, hop_length = 75, 75, NO SQUARE, log, MEAN-pooling:
results: {'mean r5': 0.10670595634078781,
'mean ap': 0.15874002830370168,
'mean p1': 0.17191011235955056}
n_ftm / hop = 32 / 16Sort of as in paper: n_ftm, hop_length = 32, 16, chroma squared, no log, MEAN-pooling:
results: {'mean r5': 0.11002302005110993,
'mean ap': 0.16368722910101047, *
'mean p1': 0.1842696629213483}
Sort of as in paper: n_ftm, hop_length = 32, 16, chroma squared, log, MEAN-pooling:
results: {'mean r5': 0.10716958135497462,
'mean ap': 0.15549415147858076,
'mean p1': 0.16179775280898875}
Sort of as in paper: n_ftm, hop_length = 32, 16, chroma squared, no log, median-pooling:
results: {'mean r5': 0.098313200532301651,
'mean ap': 0.14702795114387218,
'mean p1': 0.15730337078651685}
Sort of as in paper: n_ftm, hop_length = 32, 16, chroma squared, log, median-pooling:
results: {'mean r5': 0.086940472137101335,
'mean ap': 0.12975035440014485,
'mean p1': 0.13258426966292136}
TOP 3:
| ranking | n_ftm / hop |
expansion / compression | pooling |
|---|---|---|---|
| 1 | 75, 75 | chroma squared | mean |
| 2 | 32, 16 | log | mean |
| 3 | 32, 16 | chroma squared | mean |
In other words: best to go with mean pooling and no log.
From those options, n_ftm / hop = 75, 75 works best.
Trying one more compromise: n_ftm, hop_length = 64, 64, chroma squared, no log, MEAN-pooling:
results: {'mean r5': 0.11261683073480827,
'mean ap': 0.17040122536815025,
'mean p1': 0.18876404494382024}
| ranking | n_ftm / hop |
expansion / compression | pooling |
|---|---|---|---|
| 0 | 64, 64 | chroma squared | mean |
n_ftm = 32, stretch=[1, 2, 0.5]
(Could not use n_ftm = 64 as there are 2 songs of less than 128 beats)
results: {'mean r5': 0.1750212303302191,
'mean ap': 0.15512677446506806,
'mean p1': 0.16853932584269662}
n_ftm = 32, stretch=[1, 2]
results: {'mean r5': 0.17709617785460477,
'mean ap': 0.15597935205632796,
'mean p1': 0.16853932584269662}
n_ftm = 32, stretch=[1, 0.5]
results: {'mean r5': 0.1748866458136121,
'mean ap': 0.15156035821868111,
'mean p1': 0.16292134831460675}
n_ftm = 32, stretch=[1]
results: {'mean r5': 0.1747550452044834,
'mean ap': 0.15703052439595017,
'mean p1': 0.17078651685393259}
Conclusion: stretching doesn't help.
Not better either. Probably slower, too.
results: {'mean r5': 0.17330760999300324,
'mean ap': 0.15887497868351269,
'mean p1': 0.1842696629213483}
Conlusions: euclidean distance doesn't seem to help (though to be sure of that we should reoptimize..)
In [ ]: