SimEc and noisy input data

Previously, we've demonstrated, that SimEc can easily handle missing values in the target matrix. Here, we want to investigate how the performance of a SimEc is impacted by noise in the input feature vectors. We distinguish between noise added to the original feature vectors (i.e. noisy measurements) and feature vectors extended by several dimensions only containing noise (i.e. "big data" scenarios, where too many things were measured that are not relevant to the task). Furthermore, we experiment with random noise as well as correlated noise.

The SimEc is always trained with the more or less noisy feature vectors as input and the original similarity matrix as targets (both for a linear kernel and a class based similarity matrix). As a reference, we provide the eigendecomposition of the similarity matrix and for the linear kernel also the kernel PCA solution, which acts on only the noisy feature vectors as input and computes a linear kernel from these.

Even though kPCA can handle moderate amounts of noise fairly well, the performance nevertheless decreases significantly for larger levels of noise, especially if the noise is correlated. SimEc on the other hand succeeds in filtering out the noise.

Take home message: If you have feature vectors and a similarity matrix and you don't know how/if the similarity matrix was computed from these feature vectors, use a SimEc (or some other method that takes into account both pieces of information, e.g., compute the eigendecomposition of the similarity matrix and then train a separate regression model to learn the mapping from the feature vectors to the embedding space).



In [1]:

    
from __future__ import unicode_literals, division, print_function, absolute_import
from builtins import range
import numpy as np
np.random.seed(28)
import matplotlib.pyplot as plt
from sklearn.decomposition import KernelPCA
from sklearn.datasets import fetch_mldata
from sklearn.preprocessing import StandardScaler
import tensorflow as tf
tf.set_random_seed(28)
import keras

from simec import SimilarityEncoder
from utils import center_K, check_similarity_match
from utils_plotting import get_colors

%matplotlib inline
%load_ext autoreload
%autoreload 2









    



Using TensorFlow backend.



In [2]:

    
# load digits
mnist = fetch_mldata('MNIST original', data_home='data')
X = mnist.data/255.  # normalize to 0-1
y = np.array(mnist.target, dtype=int)
# subsample 5000 random data points
np.random.seed(42)
n_samples = 5000
n_targets = 5000
rnd_idx = np.random.permutation(X.shape[0])[:n_samples]
X, y = X[rnd_idx,:], y[rnd_idx]
# scale
ss = StandardScaler(with_std=False)
X = ss.fit_transform(X)
x_std = np.std(X)
# compute linear kernel
print("linear similarities")
K_lin = center_K(np.dot(X, X.T))
# # optimal error for linear kernel
kpca = KernelPCA(n_components=10, kernel='linear')
mse_lk_ed = check_similarity_match(kpca.fit_transform(X), K_lin)[0]
# class based similarities
print("class based similarities")
Y = np.tile(y, (len(y), 1))
S = center_K(np.array(Y==Y.T, dtype=int))
# ev embedding based on class based similarities
D, V = np.linalg.eig(S)
D1, V1 = D[np.argsort(D)[::-1]], V[:,np.argsort(D)[::-1]]
X_embed = np.dot(V1.real, np.diag(np.sqrt(np.abs(D1.real))))[:,:7]
# optimal error for class based sim
mse_class_ed = check_similarity_match(X_embed, S)[0]









    



linear similarities
class based similarities

Add random noise to input



In [3]:

    
np.random.seed(15)
noise_levels = [0., 1., 2.5, 5., 7.5, 10.]
mse_lin_kpca = []
mse_lin_simec = []
mse_class_simec = []
mse_lin_simec_pred = []
mse_class_simec_pred = []
noise_train = np.random.randn(*X.shape)
noise_train = x_std*noise_train/np.std(noise_train)
for noise_level in noise_levels:
    print("noise_level: %.1f" % noise_level)
    # add random noise to X
    X_noisy = X + noise_level*noise_train
    # lin kpca
    kpca = KernelPCA(n_components=10, kernel='linear')
    mse_lin_kpca.append(check_similarity_match(kpca.fit_transform(X_noisy), K_lin)[0])
    print("mse lin kpca: %.5f" % mse_lin_kpca[-1])
    # lin simec
    simec = SimilarityEncoder(X_noisy.shape[1], 10, n_targets, s_ll_reg=10., S_ll=K_lin[:n_targets,:n_targets],
                              l2_reg_emb=0.001, l2_reg_out=0.00001, opt=keras.optimizers.Adamax(lr=0.005))
    simec.fit(X_noisy, K_lin[:, :n_targets])
    mse_lin_simec.append(check_similarity_match(simec.transform(X_noisy), K_lin)[0])
    print("mse lin simec: %.5f" % mse_lin_simec[-1])
    mse_lin_simec_pred.append(check_similarity_match(simec.predict(X_noisy), K_lin, X_embed_is_S_approx=True)[0])
    print("mse lin simec pred: %.5f" % mse_lin_simec_pred[-1])
    # class simec
    simec = SimilarityEncoder(X_noisy.shape[1], 7, n_targets, hidden_layers=[(25, 'tanh'), (25, 'tanh')],
                              s_ll_reg=0.1, S_ll=S[:n_targets,:n_targets], orth_reg=0.00001, l2_reg=0.,
                              l2_reg_emb=0.00001, l2_reg_out=0.0000001, opt=keras.optimizers.Adamax(lr=0.01))
    simec.fit(X_noisy, S[:, :n_targets])
    mse_class_simec.append(check_similarity_match(simec.transform(X_noisy), S)[0])
    print("mse class simec: %.5f" % mse_class_simec[-1])
    mse_class_simec_pred.append(check_similarity_match(simec.predict(X_noisy), S, X_embed_is_S_approx=True)[0])
    print("mse class simec pred: %.5f" % mse_class_simec_pred[-1])
keras.backend.clear_session()
print("mse_lin_kpca=", mse_lin_kpca)
print("mse_lin_simec=", mse_lin_simec)
print("mse_class_simec=", mse_class_simec)
print("mse_lin_simec_pred=", mse_lin_simec_pred)
print("mse_class_simec_pred=", mse_class_simec_pred)
colors = get_colors(10)
plt.figure();
plt.plot([0, noise_levels[-1]], [mse_lk_ed, mse_lk_ed], '--', c=colors[4], linewidth=0.5, label='10D lin kPCA (no noise)');
plt.plot(noise_levels, mse_lin_kpca, '-o', markersize=3, c=colors[4], label='10D lin kPCA');
plt.plot(noise_levels, mse_lin_simec, '-o', markersize=3, c=colors[6], label='10D lin SimEc ($YY^{\\top}$)');
plt.plot(noise_levels, mse_lin_simec_pred, '--', markersize=3, c=colors[6], label='10D lin SimEc ($YW_l$)');
plt.legend(loc=0);
plt.title('Added random noise (linear kernel)');
plt.xticks(noise_levels, noise_levels);
plt.xlabel('Relative standard deviation of added noise')
plt.ylabel('Mean Squared Error of $\hat{S}$')
plt.figure();
plt.plot([0, noise_levels[-1]], [mse_class_ed, mse_class_ed], '--', c=colors[8], linewidth=0.5, label='7D class ED (no noise)');
plt.plot(noise_levels, mse_class_simec, '-o', markersize=3, c=colors[8], label='7D class SimEc ($YY^{\\top}$)');
plt.plot(noise_levels, mse_class_simec_pred, '--', markersize=3, c=colors[8], label='7D class SimEc ($YW_l$)');
plt.legend(loc=0);
plt.title('Added random noise (class sim)');
plt.xticks(noise_levels, noise_levels);
plt.xlabel('Relative standard deviation of added noise')
plt.ylabel('Mean Squared Error of $\hat{S}$');









    



noise_level: 0.0
mse lin kpca: 11.04889
Epoch 1/25
5000/5000 [==============================] - 3s 696us/step - loss: 930.3301
Epoch 2/25
5000/5000 [==============================] - 2s 436us/step - loss: 688.4804
Epoch 3/25
5000/5000 [==============================] - 2s 433us/step - loss: 427.1292
Epoch 4/25
5000/5000 [==============================] - 2s 434us/step - loss: 282.0368
Epoch 5/25
5000/5000 [==============================] - 2s 433us/step - loss: 204.1133
Epoch 6/25
5000/5000 [==============================] - 2s 434us/step - loss: 162.8804
Epoch 7/25
5000/5000 [==============================] - 2s 431us/step - loss: 142.1756
Epoch 8/25
5000/5000 [==============================] - 2s 437us/step - loss: 132.8677
Epoch 9/25
5000/5000 [==============================] - 2s 437us/step - loss: 128.8529
Epoch 10/25
5000/5000 [==============================] - 2s 439us/step - loss: 126.9625
Epoch 11/25
5000/5000 [==============================] - 2s 438us/step - loss: 125.9327
Epoch 12/25
5000/5000 [==============================] - 2s 437us/step - loss: 125.2007
Epoch 13/25
5000/5000 [==============================] - 2s 435us/step - loss: 124.5449
Epoch 14/25
5000/5000 [==============================] - 2s 443us/step - loss: 123.9361
Epoch 15/25
5000/5000 [==============================] - 2s 440us/step - loss: 123.4544
Epoch 16/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.1287
Epoch 17/25
5000/5000 [==============================] - 2s 438us/step - loss: 122.9618
Epoch 18/25
5000/5000 [==============================] - 2s 434us/step - loss: 122.8975
Epoch 19/25
5000/5000 [==============================] - 2s 436us/step - loss: 122.8664
Epoch 20/25
5000/5000 [==============================] - 2s 437us/step - loss: 122.8496
Epoch 21/25
5000/5000 [==============================] - 2s 440us/step - loss: 122.8517
Epoch 22/25
5000/5000 [==============================] - 2s 435us/step - loss: 122.8550
Epoch 23/25
5000/5000 [==============================] - 2s 443us/step - loss: 122.8637
Epoch 24/25
5000/5000 [==============================] - 2s 443us/step - loss: 122.8497
Epoch 25/25
5000/5000 [==============================] - 2s 440us/step - loss: 122.8512
mse lin simec: 11.10489
mse lin simec pred: 11.07954
Epoch 1/25
5000/5000 [==============================] - 4s 724us/step - loss: 0.0588
Epoch 2/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0423
Epoch 3/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0386
Epoch 4/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0362
Epoch 5/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0344
Epoch 6/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0325
Epoch 7/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0308
Epoch 8/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0290
Epoch 9/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0277
Epoch 10/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0269
Epoch 11/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0264
Epoch 12/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0260
Epoch 13/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0256
Epoch 14/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0252
Epoch 15/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0250
Epoch 16/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0247
Epoch 17/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0244
Epoch 18/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0243
Epoch 19/25
5000/5000 [==============================] - 2s 458us/step - loss: 0.0241
Epoch 20/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0241
Epoch 21/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0239
Epoch 22/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0238
Epoch 23/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0236
Epoch 24/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0234
Epoch 25/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0233
mse class simec: 0.02369
mse class simec pred: 0.02079
noise_level: 1.0
mse lin kpca: 14.64524
Epoch 1/25
5000/5000 [==============================] - 4s 703us/step - loss: 930.3086
Epoch 2/25
5000/5000 [==============================] - 2s 439us/step - loss: 684.9178
Epoch 3/25
5000/5000 [==============================] - 2s 438us/step - loss: 427.7685
Epoch 4/25
5000/5000 [==============================] - 2s 440us/step - loss: 290.3980
Epoch 5/25
5000/5000 [==============================] - 2s 438us/step - loss: 212.4717
Epoch 6/25
5000/5000 [==============================] - 2s 439us/step - loss: 171.0674
Epoch 7/25
5000/5000 [==============================] - 2s 440us/step - loss: 149.4317
Epoch 8/25
5000/5000 [==============================] - 2s 438us/step - loss: 139.4625
Epoch 9/25
5000/5000 [==============================] - 2s 440us/step - loss: 134.4570
Epoch 10/25
5000/5000 [==============================] - 2s 439us/step - loss: 131.6474
Epoch 11/25
5000/5000 [==============================] - 2s 439us/step - loss: 130.1095
Epoch 12/25
5000/5000 [==============================] - 2s 438us/step - loss: 129.2958
Epoch 13/25
5000/5000 [==============================] - 2s 436us/step - loss: 128.7896
Epoch 14/25
5000/5000 [==============================] - 2s 435us/step - loss: 128.3084
Epoch 15/25
5000/5000 [==============================] - 2s 432us/step - loss: 127.6943
Epoch 16/25
5000/5000 [==============================] - 2s 434us/step - loss: 126.8374
Epoch 17/25
5000/5000 [==============================] - 2s 436us/step - loss: 125.9411
Epoch 18/25
5000/5000 [==============================] - 2s 437us/step - loss: 125.2625
Epoch 19/25
5000/5000 [==============================] - 2s 434us/step - loss: 124.8852
Epoch 20/25
5000/5000 [==============================] - 2s 435us/step - loss: 124.7502
Epoch 21/25
5000/5000 [==============================] - 2s 436us/step - loss: 124.6855
Epoch 22/25
5000/5000 [==============================] - 2s 436us/step - loss: 124.6825
Epoch 23/25
5000/5000 [==============================] - 2s 435us/step - loss: 124.6744
Epoch 24/25
5000/5000 [==============================] - 2s 440us/step - loss: 124.6783
Epoch 25/25
5000/5000 [==============================] - 2s 433us/step - loss: 124.6840
mse lin simec: 14.18434
mse lin simec pred: 12.63906
Epoch 1/25
5000/5000 [==============================] - 4s 730us/step - loss: 0.0645
Epoch 2/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0449
Epoch 3/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0389
Epoch 4/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0352
Epoch 5/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0324
Epoch 6/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0302
Epoch 7/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0280
Epoch 8/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0261
Epoch 9/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0247
Epoch 10/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0238
Epoch 11/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0233
Epoch 12/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0229
Epoch 13/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0226
Epoch 14/25
5000/5000 [==============================] - 2s 457us/step - loss: 0.0223
Epoch 15/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0220
Epoch 16/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0219
Epoch 17/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0217
Epoch 18/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0216
Epoch 19/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0214
Epoch 20/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0213
Epoch 21/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0213
Epoch 22/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0211
Epoch 23/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0211
Epoch 24/25
5000/5000 [==============================] - 2s 459us/step - loss: 0.0210
Epoch 25/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0209
mse class simec: 0.01985
mse class simec pred: 0.01868
noise_level: 2.5
mse lin kpca: 36.67988
Epoch 1/25
5000/5000 [==============================] - 4s 702us/step - loss: 934.5202
Epoch 2/25
5000/5000 [==============================] - 2s 434us/step - loss: 684.0246
Epoch 3/25
5000/5000 [==============================] - 2s 437us/step - loss: 423.1377
Epoch 4/25
5000/5000 [==============================] - 2s 437us/step - loss: 281.9277
Epoch 5/25
5000/5000 [==============================] - 2s 441us/step - loss: 210.8691
Epoch 6/25
5000/5000 [==============================] - 2s 438us/step - loss: 169.0777
Epoch 7/25
5000/5000 [==============================] - 2s 441us/step - loss: 147.4743
Epoch 8/25
5000/5000 [==============================] - 2s 438us/step - loss: 138.2989
Epoch 9/25
5000/5000 [==============================] - 2s 437us/step - loss: 134.7942
Epoch 10/25
5000/5000 [==============================] - 2s 438us/step - loss: 133.5194
Epoch 11/25
5000/5000 [==============================] - 2s 438us/step - loss: 133.0742
Epoch 12/25
5000/5000 [==============================] - 2s 438us/step - loss: 132.8717
Epoch 13/25
5000/5000 [==============================] - 2s 439us/step - loss: 132.8355
Epoch 14/25
5000/5000 [==============================] - 2s 440us/step - loss: 132.8217
Epoch 15/25
5000/5000 [==============================] - 2s 437us/step - loss: 132.7645
Epoch 16/25
5000/5000 [==============================] - 2s 442us/step - loss: 132.7808
Epoch 17/25
5000/5000 [==============================] - 2s 440us/step - loss: 132.7586
Epoch 18/25
5000/5000 [==============================] - 2s 443us/step - loss: 132.7670
Epoch 19/25
5000/5000 [==============================] - 2s 439us/step - loss: 132.7777
Epoch 20/25
5000/5000 [==============================] - 2s 439us/step - loss: 132.7288
Epoch 21/25
5000/5000 [==============================] - 2s 439us/step - loss: 132.7151
Epoch 22/25
5000/5000 [==============================] - 2s 441us/step - loss: 132.7531
Epoch 23/25
5000/5000 [==============================] - 2s 440us/step - loss: 132.7433
Epoch 24/25
5000/5000 [==============================] - 2s 440us/step - loss: 132.7074
Epoch 25/25
5000/5000 [==============================] - 2s 440us/step - loss: 132.7787
mse lin simec: 27.01948
mse lin simec pred: 19.54427
Epoch 1/25
5000/5000 [==============================] - 4s 742us/step - loss: 0.0778
Epoch 2/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0560
Epoch 3/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0459
Epoch 4/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0401
Epoch 5/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0364
Epoch 6/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0334
Epoch 7/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0310
Epoch 8/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0289
Epoch 9/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0274
Epoch 10/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0263
Epoch 11/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0256
Epoch 12/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0251
Epoch 13/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0248
Epoch 14/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0244
Epoch 15/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0241
Epoch 16/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0240
Epoch 17/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0238
Epoch 18/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0236
Epoch 19/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0234
Epoch 20/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0233
Epoch 21/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0231
Epoch 22/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0229
Epoch 23/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0228
Epoch 24/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0227
Epoch 25/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0226
mse class simec: 0.02255
mse class simec pred: 0.02015
noise_level: 5.0
mse lin kpca: 165.51449
Epoch 1/25
5000/5000 [==============================] - 4s 714us/step - loss: 953.2848
Epoch 2/25
5000/5000 [==============================] - 2s 442us/step - loss: 736.2888
Epoch 3/25
5000/5000 [==============================] - 2s 432us/step - loss: 481.2050
Epoch 4/25
5000/5000 [==============================] - 2s 437us/step - loss: 337.4151
Epoch 5/25
5000/5000 [==============================] - 2s 439us/step - loss: 258.0240
Epoch 6/25
5000/5000 [==============================] - 2s 440us/step - loss: 207.4266
Epoch 7/25
5000/5000 [==============================] - 2s 436us/step - loss: 177.9996
Epoch 8/25
5000/5000 [==============================] - 2s 438us/step - loss: 163.1659
Epoch 9/25
5000/5000 [==============================] - 2s 438us/step - loss: 156.6184
Epoch 10/25
5000/5000 [==============================] - 2s 441us/step - loss: 154.1620
Epoch 11/25
5000/5000 [==============================] - 2s 439us/step - loss: 153.0758
Epoch 12/25
5000/5000 [==============================] - 2s 435us/step - loss: 152.7366
Epoch 13/25
5000/5000 [==============================] - 2s 439us/step - loss: 152.6326
Epoch 14/25
5000/5000 [==============================] - 2s 437us/step - loss: 152.4670
Epoch 15/25
5000/5000 [==============================] - 2s 436us/step - loss: 152.2652
Epoch 16/25
5000/5000 [==============================] - 2s 440us/step - loss: 152.2373
Epoch 17/25
5000/5000 [==============================] - 2s 437us/step - loss: 152.1527
Epoch 18/25
5000/5000 [==============================] - 2s 439us/step - loss: 152.1611
Epoch 19/25
5000/5000 [==============================] - 2s 438us/step - loss: 152.1998
Epoch 20/25
5000/5000 [==============================] - 2s 437us/step - loss: 152.4609
Epoch 21/25
5000/5000 [==============================] - 2s 437us/step - loss: 152.2751
Epoch 22/25
5000/5000 [==============================] - 2s 437us/step - loss: 152.4677
Epoch 23/25
5000/5000 [==============================] - 2s 438us/step - loss: 152.3533
Epoch 24/25
5000/5000 [==============================] - 2s 437us/step - loss: 152.2545
Epoch 25/25
5000/5000 [==============================] - 2s 434us/step - loss: 152.4019
mse lin simec: 52.71305
mse lin simec pred: 35.92962
Epoch 1/25
5000/5000 [==============================] - 4s 750us/step - loss: 0.0908
Epoch 2/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0768
Epoch 3/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0687
Epoch 4/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0621
Epoch 5/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0562
Epoch 6/25
5000/5000 [==============================] - 2s 459us/step - loss: 0.0507
Epoch 7/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0464
Epoch 8/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0427
Epoch 9/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0397
Epoch 10/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0374
Epoch 11/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0353
Epoch 12/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0338
Epoch 13/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0323
Epoch 14/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0313
Epoch 15/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0307
Epoch 16/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0301
Epoch 17/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0297
Epoch 18/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0292
Epoch 19/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0289
Epoch 20/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0285
Epoch 21/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0279
Epoch 22/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0274
Epoch 23/25
5000/5000 [==============================] - 2s 458us/step - loss: 0.0272
Epoch 24/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0268
Epoch 25/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0267
mse class simec: 0.02932
mse class simec pred: 0.02359
noise_level: 7.5
mse lin kpca: 602.60075
Epoch 1/25
5000/5000 [==============================] - 4s 725us/step - loss: 967.7128
Epoch 2/25
5000/5000 [==============================] - 2s 442us/step - loss: 792.5394
Epoch 3/25
5000/5000 [==============================] - 2s 441us/step - loss: 541.3850
Epoch 4/25
5000/5000 [==============================] - 2s 440us/step - loss: 395.2788
Epoch 5/25
5000/5000 [==============================] - 2s 443us/step - loss: 303.8807
Epoch 6/25
5000/5000 [==============================] - 2s 442us/step - loss: 254.8052
Epoch 7/25
5000/5000 [==============================] - 2s 434us/step - loss: 230.6204
Epoch 8/25
5000/5000 [==============================] - 2s 434us/step - loss: 206.4183
Epoch 9/25
5000/5000 [==============================] - 2s 434us/step - loss: 185.7300
Epoch 10/25
5000/5000 [==============================] - 2s 434us/step - loss: 176.6406
Epoch 11/25
5000/5000 [==============================] - 2s 436us/step - loss: 173.0996
Epoch 12/25
5000/5000 [==============================] - 2s 437us/step - loss: 171.8625
Epoch 13/25
5000/5000 [==============================] - 2s 436us/step - loss: 171.1562
Epoch 14/25
5000/5000 [==============================] - 2s 439us/step - loss: 170.6541
Epoch 15/25
5000/5000 [==============================] - 2s 436us/step - loss: 170.6701
Epoch 16/25
5000/5000 [==============================] - 2s 434us/step - loss: 170.6200
Epoch 17/25
5000/5000 [==============================] - 2s 439us/step - loss: 170.7883
Epoch 18/25
5000/5000 [==============================] - 2s 442us/step - loss: 170.8518
Epoch 19/25
5000/5000 [==============================] - 2s 441us/step - loss: 170.3679
Epoch 20/25
5000/5000 [==============================] - 2s 439us/step - loss: 170.4159
Epoch 21/25
5000/5000 [==============================] - 2s 443us/step - loss: 170.7265
Epoch 22/25
5000/5000 [==============================] - 2s 440us/step - loss: 170.8114
Epoch 23/25
5000/5000 [==============================] - 2s 436us/step - loss: 170.7149
Epoch 24/25
5000/5000 [==============================] - 2s 437us/step - loss: 170.6530
Epoch 25/25
5000/5000 [==============================] - 2s 436us/step - loss: 170.7973
mse lin simec: 71.21728
mse lin simec pred: 50.79828
Epoch 1/25
5000/5000 [==============================] - 4s 759us/step - loss: 0.0964
Epoch 2/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0875
Epoch 3/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0806
Epoch 4/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0752
Epoch 5/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0708
Epoch 6/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0661
Epoch 7/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0614
Epoch 8/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0563
Epoch 9/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0523
Epoch 10/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0490
Epoch 11/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0463
Epoch 12/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0444
Epoch 13/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0428
Epoch 14/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0415
Epoch 15/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0405
Epoch 16/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0396
Epoch 17/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0387
Epoch 18/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0377
Epoch 19/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0368
Epoch 20/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0362
Epoch 21/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0356
Epoch 22/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0352
Epoch 23/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0348
Epoch 24/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0341
Epoch 25/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0338
mse class simec: 0.04022
mse class simec pred: 0.03015
noise_level: 10.0
mse lin kpca: 1739.16663
Epoch 1/25
5000/5000 [==============================] - 4s 733us/step - loss: 978.2306
Epoch 2/25
5000/5000 [==============================] - 2s 438us/step - loss: 852.4168
Epoch 3/25
5000/5000 [==============================] - 2s 441us/step - loss: 617.1784
Epoch 4/25
5000/5000 [==============================] - 2s 436us/step - loss: 452.5343
Epoch 5/25
5000/5000 [==============================] - 2s 440us/step - loss: 356.8099
Epoch 6/25
5000/5000 [==============================] - 2s 441us/step - loss: 292.2287
Epoch 7/25
5000/5000 [==============================] - 2s 439us/step - loss: 258.9141
Epoch 8/25
5000/5000 [==============================] - 2s 439us/step - loss: 230.3560
Epoch 9/25
5000/5000 [==============================] - 2s 440us/step - loss: 209.4977
Epoch 10/25
5000/5000 [==============================] - 2s 441us/step - loss: 201.3093
Epoch 11/25
5000/5000 [==============================] - 2s 437us/step - loss: 194.1991
Epoch 12/25
5000/5000 [==============================] - 2s 438us/step - loss: 189.0500
Epoch 13/25
5000/5000 [==============================] - 2s 437us/step - loss: 186.7846
Epoch 14/25
5000/5000 [==============================] - 2s 435us/step - loss: 185.7554
Epoch 15/25
5000/5000 [==============================] - 2s 439us/step - loss: 186.1296
Epoch 16/25
5000/5000 [==============================] - 2s 433us/step - loss: 184.9232
Epoch 17/25
5000/5000 [==============================] - 2s 438us/step - loss: 184.8284
Epoch 18/25
5000/5000 [==============================] - 2s 446us/step - loss: 184.6793
Epoch 19/25
5000/5000 [==============================] - 2s 444us/step - loss: 185.6071
Epoch 20/25
5000/5000 [==============================] - 2s 440us/step - loss: 185.0887
Epoch 21/25
5000/5000 [==============================] - 2s 442us/step - loss: 185.2096
Epoch 22/25
5000/5000 [==============================] - 2s 438us/step - loss: 185.7253
Epoch 23/25
5000/5000 [==============================] - 2s 442us/step - loss: 185.8902
Epoch 24/25
5000/5000 [==============================] - 2s 441us/step - loss: 186.3219
Epoch 25/25
5000/5000 [==============================] - 2s 435us/step - loss: 186.0173
mse lin simec: 83.97555
mse lin simec pred: 63.87165
Epoch 1/25
5000/5000 [==============================] - 4s 768us/step - loss: 0.0985
Epoch 2/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0923
Epoch 3/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0871
Epoch 4/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0828
Epoch 5/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0787
Epoch 6/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0753
Epoch 7/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0721
Epoch 8/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0687
Epoch 9/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0655
Epoch 10/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0620
Epoch 11/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0588
Epoch 12/25
5000/5000 [==============================] - 2s 459us/step - loss: 0.0559
Epoch 13/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0537
Epoch 14/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0515
Epoch 15/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0503
Epoch 16/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0490
Epoch 17/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0477
Epoch 18/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0468
Epoch 19/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0461
Epoch 20/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0449
Epoch 21/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0442
Epoch 22/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0438
Epoch 23/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0429
Epoch 24/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0422
Epoch 25/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0412
mse class simec: 0.05039
mse class simec pred: 0.03704
mse_lin_kpca= [11.048889013360812, 14.64523906618976, 36.679884320413819, 165.51449425933259, 602.60075240526885, 1739.1666280852664]
mse_lin_simec= [11.104885667066489, 14.184343959460447, 27.019476723376446, 52.713045929809589, 71.217282035404864, 83.975551414928177]
mse_class_simec= [0.023694917837615043, 0.019853738796352213, 0.02255364477881561, 0.029319994704307746, 0.040221651146550394, 0.050387327058495031]
mse_lin_simec_pred= [11.079537061151703, 12.639056751673742, 19.544270784068193, 35.929616743527944, 50.798281679196755, 63.871651998624031]
mse_class_simec_pred= [0.020794178915291294, 0.018677734266510815, 0.020149268931628234, 0.023593546230862798, 0.030146021334916805, 0.037044586872549315]






    Out[3]:





Text(0,0.5,u'Mean Squared Error of $\\hat{S}$')

Add correlated noise to input data



In [4]:

    
np.random.seed(15)
noise_levels = [0., 1., 2.5, 5., 7.5, 10.]
mse_lin_kpca = []
mse_lin_simec = []
mse_class_simec = []
mse_lin_simec_pred = []
mse_class_simec_pred = []
# generate random covariance matrix
cov = np.random.randn(X.shape[1], X.shape[1])
cov = 0.33*(cov + cov.T + np.eye(X.shape[1]))
# make sure it's positive semidefinite
D, V = np.linalg.eig(cov)
D, V = D[np.argsort(D)[::-1]], V[:, np.argsort(D)[::-1]]
cov_embed = np.dot(V, np.diag(np.sqrt(np.abs(D))))
cov = np.dot(cov_embed[:,:100], cov_embed[:,:100].T)
# generate correlated noise data
noise_train = np.random.multivariate_normal(np.zeros(X.shape[1]), cov, X.shape[0])
noise_train = x_std*noise_train/np.std(noise_train)
for noise_level in noise_levels:
    print("noise_level: %.1f" % noise_level)
    # add random noise to X and X_test
    X_noisy = X + noise_level*noise_train
    # lin kpca
    kpca = KernelPCA(n_components=10, kernel='linear')
    mse_lin_kpca.append(check_similarity_match(kpca.fit_transform(X_noisy), K_lin)[0])
    print("mse lin kpca: %.5f" % mse_lin_kpca[-1])
    # lin simec
    simec = SimilarityEncoder(X_noisy.shape[1], 10, n_targets, s_ll_reg=10., S_ll=K_lin[:n_targets,:n_targets],
                              l2_reg_emb=0.001, l2_reg_out=0.00001, opt=keras.optimizers.Adamax(lr=0.005))
    simec.fit(X_noisy, K_lin[:, :n_targets])
    mse_lin_simec.append(check_similarity_match(simec.transform(X_noisy), K_lin)[0])
    print("mse lin simec: %.5f" % mse_lin_simec[-1])
    mse_lin_simec_pred.append(check_similarity_match(simec.predict(X_noisy), K_lin, X_embed_is_S_approx=True)[0])
    print("mse lin simec pred: %.5f" % mse_lin_simec_pred[-1])
    # class simec
    simec = SimilarityEncoder(X_noisy.shape[1], 7, n_targets, hidden_layers=[(25, 'tanh'), (25, 'tanh')],
                              s_ll_reg=0.1, S_ll=S[:n_targets,:n_targets], orth_reg=0.00001, l2_reg=0.,
                              l2_reg_emb=0.00001, l2_reg_out=0.0000001, opt=keras.optimizers.Adamax(lr=0.01))
    simec.fit(X_noisy, S[:, :n_targets])
    mse_class_simec.append(check_similarity_match(simec.transform(X_noisy), S)[0])
    print("mse class simec: %.5f" % mse_class_simec[-1])
    mse_class_simec_pred.append(check_similarity_match(simec.predict(X_noisy), S, X_embed_is_S_approx=True)[0])
    print("mse class simec pred: %.5f" % mse_class_simec_pred[-1])
keras.backend.clear_session()
print("mse_lin_kpca=", mse_lin_kpca)
print("mse_lin_simec=", mse_lin_simec)
print("mse_class_simec=", mse_class_simec)
print("mse_lin_simec_pred=", mse_lin_simec_pred)
print("mse_class_simec_pred=", mse_class_simec_pred)
colors = get_colors(10)
plt.figure();
plt.plot([0, noise_levels[-1]], [mse_lk_ed, mse_lk_ed], '--', c=colors[4], linewidth=0.5, label='10D lin kPCA (no noise)');
plt.plot(noise_levels, mse_lin_kpca, '-o', markersize=3, c=colors[4], label='10D lin kPCA');
plt.plot(noise_levels, mse_lin_simec, '-o', markersize=3, c=colors[6], label='10D lin SimEc ($YY^{\\top}$)');
plt.plot(noise_levels, mse_lin_simec_pred, '--', markersize=3, c=colors[6], label='10D lin SimEc ($YW_l$)');
plt.legend(loc=0);
plt.title('Added correlated noise (linear kernel)');
plt.xticks(noise_levels, noise_levels);
plt.xlabel('Relative standard deviation of added noise')
plt.ylabel('Mean Squared Error of $\hat{S}$')
plt.figure();
plt.plot([0, noise_levels[-1]], [mse_class_ed, mse_class_ed], '--', c=colors[8], linewidth=0.5, label='7D class ED (no noise)');
plt.plot(noise_levels, mse_class_simec, '-o', markersize=3, c=colors[8], label='7D class SimEc ($YY^{\\top}$)');
plt.plot(noise_levels, mse_class_simec_pred, '--', markersize=3, c=colors[8], label='7D class SimEc ($YW_l$)');
plt.legend(loc=0);
plt.title('Added correlated noise (class sim)');
plt.xticks(noise_levels, noise_levels);
plt.xlabel('Relative standard deviation of added noise')
plt.ylabel('Mean Squared Error of $\hat{S}$');









    



noise_level: 0.0
mse lin kpca: 11.04889
Epoch 1/25
5000/5000 [==============================] - 3s 695us/step - loss: 929.8280
Epoch 2/25
5000/5000 [==============================] - 2s 438us/step - loss: 686.2777
Epoch 3/25
5000/5000 [==============================] - 2s 433us/step - loss: 415.4617
Epoch 4/25
5000/5000 [==============================] - 2s 433us/step - loss: 271.6651
Epoch 5/25
5000/5000 [==============================] - 2s 437us/step - loss: 193.2714
Epoch 6/25
5000/5000 [==============================] - 2s 440us/step - loss: 152.9164
Epoch 7/25
5000/5000 [==============================] - 2s 438us/step - loss: 135.4061
Epoch 8/25
5000/5000 [==============================] - 2s 439us/step - loss: 128.8300
Epoch 9/25
5000/5000 [==============================] - 2s 438us/step - loss: 126.1605
Epoch 10/25
5000/5000 [==============================] - 2s 441us/step - loss: 124.8358
Epoch 11/25
5000/5000 [==============================] - 2s 442us/step - loss: 124.0413
Epoch 12/25
5000/5000 [==============================] - 2s 443us/step - loss: 123.5392
Epoch 13/25
5000/5000 [==============================] - 2s 440us/step - loss: 123.2164
Epoch 14/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.0291
Epoch 15/25
5000/5000 [==============================] - 2s 439us/step - loss: 122.9399
Epoch 16/25
5000/5000 [==============================] - 2s 438us/step - loss: 122.8798
Epoch 17/25
5000/5000 [==============================] - 2s 435us/step - loss: 122.8618
Epoch 18/25
5000/5000 [==============================] - 2s 441us/step - loss: 122.8594
Epoch 19/25
5000/5000 [==============================] - 2s 437us/step - loss: 122.8571
Epoch 20/25
5000/5000 [==============================] - 2s 437us/step - loss: 122.8493
Epoch 21/25
5000/5000 [==============================] - 2s 439us/step - loss: 122.8465
Epoch 22/25
5000/5000 [==============================] - 2s 440us/step - loss: 122.8542
Epoch 23/25
5000/5000 [==============================] - 2s 439us/step - loss: 122.8533
Epoch 24/25
5000/5000 [==============================] - 2s 443us/step - loss: 122.8559
Epoch 25/25
5000/5000 [==============================] - 2s 440us/step - loss: 122.8539
mse lin simec: 11.06487
mse lin simec pred: 11.05692
Epoch 1/25
5000/5000 [==============================] - 4s 775us/step - loss: 0.0572
Epoch 2/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0415
Epoch 3/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0384
Epoch 4/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0362
Epoch 5/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0344
Epoch 6/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0326
Epoch 7/25
5000/5000 [==============================] - 2s 458us/step - loss: 0.0307
Epoch 8/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0291
Epoch 9/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0278
Epoch 10/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0268
Epoch 11/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0262
Epoch 12/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0258
Epoch 13/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0254
Epoch 14/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0251
Epoch 15/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0247
Epoch 16/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0245
Epoch 17/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0242
Epoch 18/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0240
Epoch 19/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0238
Epoch 20/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0236
Epoch 21/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0234
Epoch 22/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0232
Epoch 23/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0231
Epoch 24/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0229
Epoch 25/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0228
mse class simec: 0.02297
mse class simec pred: 0.02023
noise_level: 1.0
mse lin kpca: 16.63035
Epoch 1/25
5000/5000 [==============================] - 4s 702us/step - loss: 929.5041
Epoch 2/25
5000/5000 [==============================] - 2s 439us/step - loss: 675.7935
Epoch 3/25
5000/5000 [==============================] - 2s 441us/step - loss: 408.2100
Epoch 4/25
5000/5000 [==============================] - 2s 439us/step - loss: 259.4934
Epoch 5/25
5000/5000 [==============================] - 2s 437us/step - loss: 182.9791
Epoch 6/25
5000/5000 [==============================] - 2s 438us/step - loss: 147.7295
Epoch 7/25
5000/5000 [==============================] - 2s 437us/step - loss: 132.6949
Epoch 8/25
5000/5000 [==============================] - 2s 439us/step - loss: 127.7024
Epoch 9/25
5000/5000 [==============================] - 2s 437us/step - loss: 125.8479
Epoch 10/25
5000/5000 [==============================] - 2s 438us/step - loss: 124.8955
Epoch 11/25
5000/5000 [==============================] - 2s 438us/step - loss: 124.2347
Epoch 12/25
5000/5000 [==============================] - 2s 441us/step - loss: 123.7676
Epoch 13/25
5000/5000 [==============================] - 2s 437us/step - loss: 123.4358
Epoch 14/25
5000/5000 [==============================] - 2s 440us/step - loss: 123.2036
Epoch 15/25
5000/5000 [==============================] - 2s 437us/step - loss: 123.0525
Epoch 16/25
5000/5000 [==============================] - 2s 440us/step - loss: 122.9666
Epoch 17/25
5000/5000 [==============================] - 2s 434us/step - loss: 122.9509
Epoch 18/25
5000/5000 [==============================] - 2s 434us/step - loss: 122.8980
Epoch 19/25
5000/5000 [==============================] - 2s 434us/step - loss: 122.9313
Epoch 20/25
5000/5000 [==============================] - 2s 436us/step - loss: 122.9087
Epoch 21/25
5000/5000 [==============================] - 2s 437us/step - loss: 122.9027
Epoch 22/25
5000/5000 [==============================] - 2s 431us/step - loss: 122.8988
Epoch 23/25
5000/5000 [==============================] - 2s 440us/step - loss: 122.9232
Epoch 24/25
5000/5000 [==============================] - 2s 438us/step - loss: 122.8897
Epoch 25/25
5000/5000 [==============================] - 2s 441us/step - loss: 122.9026
mse lin simec: 11.19628
mse lin simec pred: 11.12228
Epoch 1/25
5000/5000 [==============================] - 4s 733us/step - loss: 0.0636
Epoch 2/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0443
Epoch 3/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0399
Epoch 4/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0371
Epoch 5/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0351
Epoch 6/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0330
Epoch 7/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0313
Epoch 8/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0296
Epoch 9/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0282
Epoch 10/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0271
Epoch 11/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0263
Epoch 12/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0256
Epoch 13/25
5000/5000 [==============================] - 2s 459us/step - loss: 0.0251
Epoch 14/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0247
Epoch 15/25
5000/5000 [==============================] - 2s 459us/step - loss: 0.0245
Epoch 16/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0242
Epoch 17/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0239
Epoch 18/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0236
Epoch 19/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0234
Epoch 20/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0233
Epoch 21/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0231
Epoch 22/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0230
Epoch 23/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0228
Epoch 24/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0227
Epoch 25/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0226
mse class simec: 0.02239
mse class simec pred: 0.02009
noise_level: 2.5
mse lin kpca: 233.59581
Epoch 1/25
5000/5000 [==============================] - 4s 710us/step - loss: 930.8435
Epoch 2/25
5000/5000 [==============================] - 2s 438us/step - loss: 666.2876
Epoch 3/25
5000/5000 [==============================] - 2s 435us/step - loss: 408.1970
Epoch 4/25
5000/5000 [==============================] - 2s 443us/step - loss: 266.7942
Epoch 5/25
5000/5000 [==============================] - 2s 442us/step - loss: 188.9464
Epoch 6/25
5000/5000 [==============================] - 2s 441us/step - loss: 152.4086
Epoch 7/25
5000/5000 [==============================] - 2s 439us/step - loss: 136.4301
Epoch 8/25
5000/5000 [==============================] - 2s 442us/step - loss: 129.1088
Epoch 9/25
5000/5000 [==============================] - 2s 437us/step - loss: 125.8796
Epoch 10/25
5000/5000 [==============================] - 2s 438us/step - loss: 124.4312
Epoch 11/25
5000/5000 [==============================] - 2s 437us/step - loss: 123.6993
Epoch 12/25
5000/5000 [==============================] - 2s 440us/step - loss: 123.3720
Epoch 13/25
5000/5000 [==============================] - 2s 441us/step - loss: 123.3088
Epoch 14/25
5000/5000 [==============================] - 2s 437us/step - loss: 123.1124
Epoch 15/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.1083
Epoch 16/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.1251
Epoch 17/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.0668
Epoch 18/25
5000/5000 [==============================] - 2s 441us/step - loss: 123.1688
Epoch 19/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.1249
Epoch 20/25
5000/5000 [==============================] - 2s 438us/step - loss: 123.0689
Epoch 21/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.1103
Epoch 22/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.1762
Epoch 23/25
5000/5000 [==============================] - 2s 442us/step - loss: 123.0727
Epoch 24/25
5000/5000 [==============================] - 2s 442us/step - loss: 123.0892
Epoch 25/25
5000/5000 [==============================] - 2s 437us/step - loss: 123.1080
mse lin simec: 11.36421
mse lin simec pred: 11.20780
Epoch 1/25
5000/5000 [==============================] - 4s 740us/step - loss: 0.0773
Epoch 2/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0552
Epoch 3/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0459
Epoch 4/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0418
Epoch 5/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0388
Epoch 6/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0367
Epoch 7/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0346
Epoch 8/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0325
Epoch 9/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0308
Epoch 10/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0292
Epoch 11/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0284
Epoch 12/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0277
Epoch 13/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0273
Epoch 14/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0269
Epoch 15/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0265
Epoch 16/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0264
Epoch 17/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0259
Epoch 18/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0257
Epoch 19/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0255
Epoch 20/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0253
Epoch 21/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0249
Epoch 22/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0249
Epoch 23/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0248
Epoch 24/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0244
Epoch 25/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0242
mse class simec: 0.02533
mse class simec pred: 0.02141
noise_level: 5.0
mse lin kpca: 3160.44334
Epoch 1/25
5000/5000 [==============================] - 4s 715us/step - loss: 944.6172
Epoch 2/25
5000/5000 [==============================] - 2s 438us/step - loss: 691.1680
Epoch 3/25
5000/5000 [==============================] - 2s 439us/step - loss: 435.0019
Epoch 4/25
5000/5000 [==============================] - 2s 440us/step - loss: 292.4597
Epoch 5/25
5000/5000 [==============================] - 2s 435us/step - loss: 212.3072
Epoch 6/25
5000/5000 [==============================] - 2s 438us/step - loss: 163.0008
Epoch 7/25
5000/5000 [==============================] - 2s 439us/step - loss: 139.4617
Epoch 8/25
5000/5000 [==============================] - 2s 440us/step - loss: 129.8371
Epoch 9/25
5000/5000 [==============================] - 2s 439us/step - loss: 126.4630
Epoch 10/25
5000/5000 [==============================] - 2s 438us/step - loss: 124.8020
Epoch 11/25
5000/5000 [==============================] - 2s 439us/step - loss: 124.4136
Epoch 12/25
5000/5000 [==============================] - 2s 442us/step - loss: 123.8622
Epoch 13/25
5000/5000 [==============================] - 2s 440us/step - loss: 124.0572
Epoch 14/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.8631
Epoch 15/25
5000/5000 [==============================] - 2s 438us/step - loss: 123.7512
Epoch 16/25
5000/5000 [==============================] - 2s 438us/step - loss: 123.7558
Epoch 17/25
5000/5000 [==============================] - 2s 437us/step - loss: 123.6922
Epoch 18/25
5000/5000 [==============================] - 2s 435us/step - loss: 124.0873
Epoch 19/25
5000/5000 [==============================] - 2s 436us/step - loss: 123.6251
Epoch 20/25
5000/5000 [==============================] - 2s 438us/step - loss: 123.5307
Epoch 21/25
5000/5000 [==============================] - 2s 437us/step - loss: 123.8163
Epoch 22/25
5000/5000 [==============================] - 2s 440us/step - loss: 123.6623
Epoch 23/25
5000/5000 [==============================] - 2s 436us/step - loss: 124.1400
Epoch 24/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.5780
Epoch 25/25
5000/5000 [==============================] - 2s 438us/step - loss: 123.7393
mse lin simec: 12.71344
mse lin simec pred: 11.88189
Epoch 1/25
5000/5000 [==============================] - 4s 749us/step - loss: 0.0914
Epoch 2/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0788
Epoch 3/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0693
Epoch 4/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0606
Epoch 5/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0537
Epoch 6/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0485
Epoch 7/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0455
Epoch 8/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0430
Epoch 9/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0403
Epoch 10/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0380
Epoch 11/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0365
Epoch 12/25
5000/5000 [==============================] - 2s 459us/step - loss: 0.0352
Epoch 13/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0343
Epoch 14/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0333
Epoch 15/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0328
Epoch 16/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0322
Epoch 17/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0322
Epoch 18/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0320
Epoch 19/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0310
Epoch 20/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0311
Epoch 21/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0307
Epoch 22/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0303
Epoch 23/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0300
Epoch 24/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0298
Epoch 25/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0297
mse class simec: 0.03364
mse class simec pred: 0.02609
noise_level: 7.5
mse lin kpca: 15565.70460
Epoch 1/25
5000/5000 [==============================] - 4s 728us/step - loss: 959.7544
Epoch 2/25
5000/5000 [==============================] - 2s 442us/step - loss: 733.5773
Epoch 3/25
5000/5000 [==============================] - 2s 440us/step - loss: 462.8426
Epoch 4/25
5000/5000 [==============================] - 2s 440us/step - loss: 311.4949
Epoch 5/25
5000/5000 [==============================] - 2s 436us/step - loss: 226.1826
Epoch 6/25
5000/5000 [==============================] - 2s 438us/step - loss: 177.3704
Epoch 7/25
5000/5000 [==============================] - 2s 439us/step - loss: 144.0050
Epoch 8/25
5000/5000 [==============================] - 2s 442us/step - loss: 130.3659
Epoch 9/25
5000/5000 [==============================] - 2s 443us/step - loss: 126.3138
Epoch 10/25
5000/5000 [==============================] - 2s 444us/step - loss: 126.2389
Epoch 11/25
5000/5000 [==============================] - 2s 441us/step - loss: 125.3723
Epoch 12/25
5000/5000 [==============================] - 2s 436us/step - loss: 125.2548
Epoch 13/25
5000/5000 [==============================] - 2s 441us/step - loss: 125.1501
Epoch 14/25
5000/5000 [==============================] - 2s 439us/step - loss: 125.1200
Epoch 15/25
5000/5000 [==============================] - 2s 440us/step - loss: 125.1133
Epoch 16/25
5000/5000 [==============================] - 2s 444us/step - loss: 124.9595
Epoch 17/25
5000/5000 [==============================] - 2s 444us/step - loss: 125.3277
Epoch 18/25
5000/5000 [==============================] - 2s 441us/step - loss: 125.2467
Epoch 19/25
5000/5000 [==============================] - 2s 441us/step - loss: 124.8858
Epoch 20/25
5000/5000 [==============================] - 2s 440us/step - loss: 125.1009
Epoch 21/25
5000/5000 [==============================] - 2s 442us/step - loss: 124.7413
Epoch 22/25
5000/5000 [==============================] - 2s 440us/step - loss: 125.0465
Epoch 23/25
5000/5000 [==============================] - 2s 443us/step - loss: 124.9619
Epoch 24/25
5000/5000 [==============================] - 2s 439us/step - loss: 124.9481
Epoch 25/25
5000/5000 [==============================] - 2s 442us/step - loss: 124.9758
mse lin simec: 14.83895
mse lin simec pred: 12.85276
Epoch 1/25
5000/5000 [==============================] - 4s 761us/step - loss: 0.0981
Epoch 2/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0929
Epoch 3/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0881
Epoch 4/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0842
Epoch 5/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0796
Epoch 6/25
5000/5000 [==============================] - 2s 474us/step - loss: 0.0742
Epoch 7/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0702
Epoch 8/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0660
Epoch 9/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0622
Epoch 10/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0590
Epoch 11/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0553
Epoch 12/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0526
Epoch 13/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0499
Epoch 14/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0484
Epoch 15/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0468
Epoch 16/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0461
Epoch 17/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0449
Epoch 18/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0441
Epoch 19/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0433
Epoch 20/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0429
Epoch 21/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0420
Epoch 22/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0408
Epoch 23/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0396
Epoch 24/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0400
Epoch 25/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0394
mse class simec: 0.04663
mse class simec pred: 0.03444
noise_level: 10.0
mse lin kpca: 48926.11684
Epoch 1/25
5000/5000 [==============================] - 4s 733us/step - loss: 976.2950
Epoch 2/25
5000/5000 [==============================] - 2s 438us/step - loss: 792.9485
Epoch 3/25
5000/5000 [==============================] - 2s 439us/step - loss: 514.3168
Epoch 4/25
5000/5000 [==============================] - 2s 440us/step - loss: 332.8432
Epoch 5/25
5000/5000 [==============================] - 2s 440us/step - loss: 243.5296
Epoch 6/25
5000/5000 [==============================] - 2s 440us/step - loss: 201.6492
Epoch 7/25
5000/5000 [==============================] - 2s 437us/step - loss: 173.2257
Epoch 8/25
5000/5000 [==============================] - 2s 435us/step - loss: 152.9615
Epoch 9/25
5000/5000 [==============================] - 2s 436us/step - loss: 143.3784
Epoch 10/25
5000/5000 [==============================] - 2s 439us/step - loss: 136.0203
Epoch 11/25
5000/5000 [==============================] - 2s 441us/step - loss: 132.0758
Epoch 12/25
5000/5000 [==============================] - 2s 438us/step - loss: 131.7019
Epoch 13/25
5000/5000 [==============================] - 2s 441us/step - loss: 129.5281
Epoch 14/25
5000/5000 [==============================] - 2s 438us/step - loss: 128.5877
Epoch 15/25
5000/5000 [==============================] - 2s 439us/step - loss: 129.0755
Epoch 16/25
5000/5000 [==============================] - 2s 441us/step - loss: 127.3996
Epoch 17/25
5000/5000 [==============================] - 2s 439us/step - loss: 128.2786
Epoch 18/25
5000/5000 [==============================] - 2s 444us/step - loss: 127.3363
Epoch 19/25
5000/5000 [==============================] - 2s 441us/step - loss: 125.5833
Epoch 20/25
5000/5000 [==============================] - 2s 439us/step - loss: 126.5229
Epoch 21/25
5000/5000 [==============================] - 2s 439us/step - loss: 127.0562
Epoch 22/25
5000/5000 [==============================] - 2s 436us/step - loss: 126.4579
Epoch 23/25
5000/5000 [==============================] - 2s 440us/step - loss: 126.5536
Epoch 24/25
5000/5000 [==============================] - 2s 437us/step - loss: 125.0527
Epoch 25/25
5000/5000 [==============================] - 2s 435us/step - loss: 126.0810
mse lin simec: 17.74538
mse lin simec pred: 14.21968
Epoch 1/25
5000/5000 [==============================] - 4s 768us/step - loss: 0.0994
Epoch 2/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0974
Epoch 3/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0944
Epoch 4/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0918
Epoch 5/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0896
Epoch 6/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0879
Epoch 7/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0862
Epoch 8/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0843
Epoch 9/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0823
Epoch 10/25
5000/5000 [==============================] - 2s 457us/step - loss: 0.0792
Epoch 11/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0757
Epoch 12/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0724
Epoch 13/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0694
Epoch 14/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0672
Epoch 15/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0651
Epoch 16/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0633
Epoch 17/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0618
Epoch 18/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0600
Epoch 19/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0589
Epoch 20/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0572
Epoch 21/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0567
Epoch 22/25
5000/5000 [==============================] - 2s 473us/step - loss: 0.0551
Epoch 23/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0545
Epoch 24/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0536
Epoch 25/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0523
mse class simec: 0.06213
mse class simec pred: 0.04667
mse_lin_kpca= [11.048889013360812, 16.630349983403203, 233.59581191563666, 3160.4433397112712, 15565.704600665222, 48926.116838197064]
mse_lin_simec= [11.064870549307846, 11.19628183407905, 11.364212517738833, 12.713435047399559, 14.83894990360009, 17.745377097358837]
mse_class_simec= [0.022973106029298543, 0.022392167805749687, 0.02532965950572617, 0.03363502232853445, 0.046625164226089685, 0.062134365293436349]
mse_lin_simec_pred= [11.056919617229161, 11.122283925543909, 11.207797092435086, 11.881886983455374, 12.852755415807895, 14.21967985871825]
mse_class_simec_pred= [0.020233354887419654, 0.020090282902607993, 0.021414153711094633, 0.026085849989466756, 0.034442757999032174, 0.046672728791286089]






    Out[4]:





Text(0,0.5,u'Mean Squared Error of $\\hat{S}$')

Add additional input dimensions with random noise



In [5]:

    
np.random.seed(15)
noise_levels = [0, X.shape[1], 2*X.shape[1], 4*X.shape[1], 6*X.shape[1], 8*X.shape[1], 10*X.shape[1]]
mse_lin_kpca = []
mse_lin_simec = []
mse_class_simec = []
mse_lin_simec_pred = []
mse_class_simec_pred = []
noise_train = np.random.randn(X.shape[0], max(noise_levels))
noise_train = x_std*noise_train/np.std(noise_train)
for noise_level in noise_levels:
    print("noise dimensions: %i" % noise_level)
    # add random noise as additional feature dimensions
    X_noisy = np.hstack((X, noise_train[:,:noise_level]))
    # lin kpca
    kpca = KernelPCA(n_components=10, kernel='linear')
    mse_lin_kpca.append(check_similarity_match(kpca.fit_transform(X_noisy), K_lin)[0])
    print("mse lin kpca: %.5f" % mse_lin_kpca[-1])
    # lin simec
    simec = SimilarityEncoder(X_noisy.shape[1], 10, n_targets, s_ll_reg=10., S_ll=K_lin[:n_targets,:n_targets],
                              l2_reg_emb=0.001, l2_reg_out=0.00001, opt=keras.optimizers.Adamax(lr=0.005 if noise_level <= 4*X.shape[1] else 0.001))
    simec.fit(X_noisy, K_lin[:, :n_targets])
    mse_lin_simec.append(check_similarity_match(simec.transform(X_noisy), K_lin)[0])
    print("mse lin simec: %.5f" % mse_lin_simec[-1])
    mse_lin_simec_pred.append(check_similarity_match(simec.predict(X_noisy), K_lin, X_embed_is_S_approx=True)[0])
    print("mse lin simec pred: %.5f" % mse_lin_simec_pred[-1])
    # class simec
    simec = SimilarityEncoder(X_noisy.shape[1], 7, n_targets, hidden_layers=[(25, 'tanh'), (25, 'tanh')],
                              s_ll_reg=0.1, S_ll=S[:n_targets,:n_targets], orth_reg=0.00001, l2_reg=0.,
                              l2_reg_emb=0.00001, l2_reg_out=0.0000001, opt=keras.optimizers.Adamax(lr=0.01 if noise_level < 4*X.shape[1] else 0.005))
    simec.fit(X_noisy, S[:, :n_targets])
    mse_class_simec.append(check_similarity_match(simec.transform(X_noisy), S)[0])
    print("mse class simec: %.5f" % mse_class_simec[-1])
    mse_class_simec_pred.append(check_similarity_match(simec.predict(X_noisy), S, X_embed_is_S_approx=True)[0])
    print("mse class simec pred: %.5f" % mse_class_simec_pred[-1])
keras.backend.clear_session()
print("mse_lin_kpca=", mse_lin_kpca)
print("mse_lin_simec=", mse_lin_simec)
print("mse_class_simec=", mse_class_simec)
print("mse_lin_simec_pred=", mse_lin_simec_pred)
print("mse_class_simec_pred=", mse_class_simec_pred)
colors = get_colors(10)
plt.figure();
plt.plot([0, noise_levels[-1]], [mse_lk_ed, mse_lk_ed], '--', c=colors[4], linewidth=0.5, label='10D lin kPCA (no noise)');
plt.plot(noise_levels, mse_lin_kpca, '-o', markersize=3, c=colors[4], label='10D lin kPCA');
plt.plot(noise_levels, mse_lin_simec, '-o', markersize=3, c=colors[6], label='10D lin SimEc ($YY^{\\top}$)');
plt.plot(noise_levels, mse_lin_simec_pred, '--', markersize=3, c=colors[6], label='10D lin SimEc ($YW_l$)');
plt.legend(loc=0);
plt.title('Added dimensions with random noise (linear kernel)');
plt.xticks(noise_levels, [0, '$D$', '$2\\times D$', '$4\\times D$', '$6\\times D$', '$8\\times D$', '$10\\times D$']);
plt.xlabel('Additional noise dimensions (where $D = 784$)')
plt.ylabel('Mean Squared Error of $\hat{S}$')
plt.figure();
plt.plot([0, noise_levels[-1]], [mse_class_ed, mse_class_ed], '--', c=colors[8], linewidth=0.5, label='7D class ED (no noise)');
plt.plot(noise_levels, mse_class_simec, '-o', markersize=3, c=colors[8], label='7D class SimEc ($YY^{\\top}$)');
plt.plot(noise_levels, mse_class_simec_pred, '--', markersize=3, c=colors[8], label='7D class SimEc ($YW_l$)');
plt.legend(loc=0);
plt.title('Added dimensions with random noise (class sim)');
plt.xticks(noise_levels, [0, '$D$', '$2\\times D$', '$4\\times D$', '$6\\times D$', '$8\\times D$', '$10\\times D$']);
plt.xlabel('Additional noise dimensions (where $D = 784$)');
plt.ylabel('Mean Squared Error of $\hat{S}$');









    



noise dimensions: 0
mse lin kpca: 11.04889
Epoch 1/25
5000/5000 [==============================] - 4s 742us/step - loss: 930.5728
Epoch 2/25
5000/5000 [==============================] - 2s 437us/step - loss: 689.9882
Epoch 3/25
5000/5000 [==============================] - 2s 438us/step - loss: 415.4672
Epoch 4/25
5000/5000 [==============================] - 2s 437us/step - loss: 271.9111
Epoch 5/25
5000/5000 [==============================] - 2s 454us/step - loss: 200.3961
Epoch 6/25
5000/5000 [==============================] - 2s 463us/step - loss: 162.3277
Epoch 7/25
5000/5000 [==============================] - 2s 448us/step - loss: 139.2309
Epoch 8/25
5000/5000 [==============================] - 2s 450us/step - loss: 130.0155
Epoch 9/25
5000/5000 [==============================] - 2s 462us/step - loss: 126.2490
Epoch 10/25
5000/5000 [==============================] - 2s 444us/step - loss: 124.4264
Epoch 11/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.5253
Epoch 12/25
5000/5000 [==============================] - 2s 447us/step - loss: 123.1138
Epoch 13/25
5000/5000 [==============================] - 2s 436us/step - loss: 122.9468
Epoch 14/25
5000/5000 [==============================] - 2s 437us/step - loss: 122.8828
Epoch 15/25
5000/5000 [==============================] - 2s 438us/step - loss: 122.8620
Epoch 16/25
5000/5000 [==============================] - 2s 452us/step - loss: 122.8559
Epoch 17/25
5000/5000 [==============================] - 2s 456us/step - loss: 122.8572
Epoch 18/25
5000/5000 [==============================] - 2s 453us/step - loss: 122.8538
Epoch 19/25
5000/5000 [==============================] - 2s 430us/step - loss: 122.8517
Epoch 20/25
5000/5000 [==============================] - 2s 448us/step - loss: 122.8568
Epoch 21/25
5000/5000 [==============================] - 2s 456us/step - loss: 122.8586
Epoch 22/25
5000/5000 [==============================] - 2s 443us/step - loss: 122.8488
Epoch 23/25
5000/5000 [==============================] - 2s 446us/step - loss: 122.8460
Epoch 24/25
5000/5000 [==============================] - 2s 443us/step - loss: 122.8533
Epoch 25/25
5000/5000 [==============================] - 2s 432us/step - loss: 122.8568
mse lin simec: 11.07253
mse lin simec pred: 11.06152
Epoch 1/25
5000/5000 [==============================] - 4s 779us/step - loss: 0.0589
Epoch 2/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0426
Epoch 3/25
5000/5000 [==============================] - 2s 479us/step - loss: 0.0389
Epoch 4/25
5000/5000 [==============================] - 2s 484us/step - loss: 0.0366
Epoch 5/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0347
Epoch 6/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0330
Epoch 7/25
5000/5000 [==============================] - 2s 483us/step - loss: 0.0312
Epoch 8/25
5000/5000 [==============================] - 2s 476us/step - loss: 0.0295
Epoch 9/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0281
Epoch 10/25
5000/5000 [==============================] - 2s 475us/step - loss: 0.0269
Epoch 11/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0262
Epoch 12/25
5000/5000 [==============================] - 2s 480us/step - loss: 0.0257
Epoch 13/25
5000/5000 [==============================] - 2s 475us/step - loss: 0.0252
Epoch 14/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0249
Epoch 15/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0246
Epoch 16/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0244
Epoch 17/25
5000/5000 [==============================] - 2s 479us/step - loss: 0.0241
Epoch 18/25
5000/5000 [==============================] - 2s 481us/step - loss: 0.0240
Epoch 19/25
5000/5000 [==============================] - 2s 483us/step - loss: 0.0238
Epoch 20/25
5000/5000 [==============================] - 2s 475us/step - loss: 0.0236
Epoch 21/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0233
Epoch 22/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0232
Epoch 23/25
5000/5000 [==============================] - 2s 475us/step - loss: 0.0231
Epoch 24/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0230
Epoch 25/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0227
mse class simec: 0.02285
mse class simec pred: 0.02027
noise dimensions: 784
mse lin kpca: 11.06464
Epoch 1/25
5000/5000 [==============================] - 4s 769us/step - loss: 930.4807
Epoch 2/25
5000/5000 [==============================] - 2s 439us/step - loss: 688.2385
Epoch 3/25
5000/5000 [==============================] - 2s 446us/step - loss: 419.7593
Epoch 4/25
5000/5000 [==============================] - 2s 440us/step - loss: 267.8033
Epoch 5/25
5000/5000 [==============================] - 2s 437us/step - loss: 186.3456
Epoch 6/25
5000/5000 [==============================] - 2s 440us/step - loss: 150.5180
Epoch 7/25
5000/5000 [==============================] - 2s 440us/step - loss: 136.2151
Epoch 8/25
5000/5000 [==============================] - 2s 436us/step - loss: 130.0111
Epoch 9/25
5000/5000 [==============================] - 2s 438us/step - loss: 127.2332
Epoch 10/25
5000/5000 [==============================] - 2s 436us/step - loss: 125.8639
Epoch 11/25
5000/5000 [==============================] - 2s 430us/step - loss: 125.0079
Epoch 12/25
5000/5000 [==============================] - 2s 437us/step - loss: 124.3738
Epoch 13/25
5000/5000 [==============================] - 2s 436us/step - loss: 123.8587
Epoch 14/25
5000/5000 [==============================] - 2s 449us/step - loss: 123.4781
Epoch 15/25
5000/5000 [==============================] - 2s 435us/step - loss: 123.2253
Epoch 16/25
5000/5000 [==============================] - 2s 434us/step - loss: 123.0588
Epoch 17/25
5000/5000 [==============================] - 2s 438us/step - loss: 123.0237
Epoch 18/25
5000/5000 [==============================] - 2s 431us/step - loss: 122.9640
Epoch 19/25
5000/5000 [==============================] - 2s 430us/step - loss: 122.9579
Epoch 20/25
5000/5000 [==============================] - 2s 430us/step - loss: 122.9504
Epoch 21/25
5000/5000 [==============================] - 2s 434us/step - loss: 122.9761
Epoch 22/25
5000/5000 [==============================] - 2s 433us/step - loss: 122.9678
Epoch 23/25
5000/5000 [==============================] - 2s 434us/step - loss: 122.9272
Epoch 24/25
5000/5000 [==============================] - 2s 434us/step - loss: 122.9471
Epoch 25/25
5000/5000 [==============================] - 2s 435us/step - loss: 122.9507
mse lin simec: 11.32595
mse lin simec pred: 11.17770
Epoch 1/25
5000/5000 [==============================] - 4s 810us/step - loss: 0.0604
Epoch 2/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0416
Epoch 3/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0359
Epoch 4/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0327
Epoch 5/25
5000/5000 [==============================] - 2s 483us/step - loss: 0.0302
Epoch 6/25
5000/5000 [==============================] - 2s 475us/step - loss: 0.0283
Epoch 7/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0263
Epoch 8/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0245
Epoch 9/25
5000/5000 [==============================] - 2s 479us/step - loss: 0.0233
Epoch 10/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0226
Epoch 11/25
5000/5000 [==============================] - 2s 474us/step - loss: 0.0221
Epoch 12/25
5000/5000 [==============================] - 2s 474us/step - loss: 0.0219
Epoch 13/25
5000/5000 [==============================] - 2s 486us/step - loss: 0.0218
Epoch 14/25
5000/5000 [==============================] - 2s 473us/step - loss: 0.0216
Epoch 15/25
5000/5000 [==============================] - 2s 490us/step - loss: 0.0214
Epoch 16/25
5000/5000 [==============================] - 2s 490us/step - loss: 0.0213
Epoch 17/25
5000/5000 [==============================] - 2s 496us/step - loss: 0.0211
Epoch 18/25
5000/5000 [==============================] - 2s 486us/step - loss: 0.0210
Epoch 19/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0210
Epoch 20/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0209
Epoch 21/25
5000/5000 [==============================] - 2s 486us/step - loss: 0.0208
Epoch 22/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0208
Epoch 23/25
5000/5000 [==============================] - 2s 479us/step - loss: 0.0207
Epoch 24/25
5000/5000 [==============================] - 2s 479us/step - loss: 0.0206
Epoch 25/25
5000/5000 [==============================] - 2s 473us/step - loss: 0.0206
mse class simec: 0.01923
mse class simec pred: 0.01831
noise dimensions: 1568
mse lin kpca: 11.08361
Epoch 1/25
5000/5000 [==============================] - 4s 777us/step - loss: 929.0635
Epoch 2/25
5000/5000 [==============================] - 2s 454us/step - loss: 678.6342
Epoch 3/25
5000/5000 [==============================] - 2s 451us/step - loss: 411.6919
Epoch 4/25
5000/5000 [==============================] - 2s 442us/step - loss: 267.0230
Epoch 5/25
5000/5000 [==============================] - 2s 436us/step - loss: 188.6181
Epoch 6/25
5000/5000 [==============================] - 2s 437us/step - loss: 151.4844
Epoch 7/25
5000/5000 [==============================] - 2s 441us/step - loss: 135.6772
Epoch 8/25
5000/5000 [==============================] - 2s 440us/step - loss: 129.7710
Epoch 9/25
5000/5000 [==============================] - 2s 460us/step - loss: 127.4398
Epoch 10/25
5000/5000 [==============================] - 2s 453us/step - loss: 126.5054
Epoch 11/25
5000/5000 [==============================] - 2s 433us/step - loss: 125.8518
Epoch 12/25
5000/5000 [==============================] - 2s 440us/step - loss: 125.1666
Epoch 13/25
5000/5000 [==============================] - 2s 445us/step - loss: 124.5969
Epoch 14/25
5000/5000 [==============================] - 2s 447us/step - loss: 124.0593
Epoch 15/25
5000/5000 [==============================] - 2s 441us/step - loss: 123.6686
Epoch 16/25
5000/5000 [==============================] - 2s 440us/step - loss: 123.4107
Epoch 17/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.2226
Epoch 18/25
5000/5000 [==============================] - 2s 436us/step - loss: 123.1845
Epoch 19/25
5000/5000 [==============================] - 2s 429us/step - loss: 123.1871
Epoch 20/25
5000/5000 [==============================] - 2s 433us/step - loss: 123.1775
Epoch 21/25
5000/5000 [==============================] - 2s 424us/step - loss: 123.1608
Epoch 22/25
5000/5000 [==============================] - 2s 426us/step - loss: 123.1275
Epoch 23/25
5000/5000 [==============================] - 2s 431us/step - loss: 123.1510
Epoch 24/25
5000/5000 [==============================] - 2s 424us/step - loss: 123.1603
Epoch 25/25
5000/5000 [==============================] - 2s 429us/step - loss: 123.1646
mse lin simec: 11.63544
mse lin simec pred: 11.33875
Epoch 1/25
5000/5000 [==============================] - 4s 796us/step - loss: 0.0633
Epoch 2/25
5000/5000 [==============================] - 2s 446us/step - loss: 0.0395
Epoch 3/25
5000/5000 [==============================] - 2s 446us/step - loss: 0.0330
Epoch 4/25
5000/5000 [==============================] - 2s 451us/step - loss: 0.0299
Epoch 5/25
5000/5000 [==============================] - 2s 444us/step - loss: 0.0278
Epoch 6/25
5000/5000 [==============================] - 2s 449us/step - loss: 0.0259
Epoch 7/25
5000/5000 [==============================] - 2s 458us/step - loss: 0.0243
Epoch 8/25
5000/5000 [==============================] - 2s 455us/step - loss: 0.0229
Epoch 9/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0221
Epoch 10/25
5000/5000 [==============================] - 2s 496us/step - loss: 0.0217
Epoch 11/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0215
Epoch 12/25
5000/5000 [==============================] - 2s 457us/step - loss: 0.0213
Epoch 13/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0211
Epoch 14/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0210
Epoch 15/25
5000/5000 [==============================] - 2s 459us/step - loss: 0.0209
Epoch 16/25
5000/5000 [==============================] - 2s 480us/step - loss: 0.0208
Epoch 17/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0207
Epoch 18/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0206
Epoch 19/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0206
Epoch 20/25
5000/5000 [==============================] - 2s 458us/step - loss: 0.0205
Epoch 21/25
5000/5000 [==============================] - 2s 453us/step - loss: 0.0205
Epoch 22/25
5000/5000 [==============================] - 2s 454us/step - loss: 0.0204
Epoch 23/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0204
Epoch 24/25
5000/5000 [==============================] - 2s 454us/step - loss: 0.0204
Epoch 25/25
5000/5000 [==============================] - 2s 458us/step - loss: 0.0203
mse class simec: 0.01888
mse class simec pred: 0.01810
noise dimensions: 3136
mse lin kpca: 11.13002
Epoch 1/25
5000/5000 [==============================] - 4s 848us/step - loss: 991.4211
Epoch 2/25
5000/5000 [==============================] - 2s 448us/step - loss: 959.7998
Epoch 3/25
5000/5000 [==============================] - 2s 450us/step - loss: 927.8330
Epoch 4/25
5000/5000 [==============================] - 2s 463us/step - loss: 894.5039
Epoch 5/25
5000/5000 [==============================] - 2s 461us/step - loss: 850.5942
Epoch 6/25
5000/5000 [==============================] - 2s 441us/step - loss: 796.9009
Epoch 7/25
5000/5000 [==============================] - 2s 439us/step - loss: 735.7746
Epoch 8/25
5000/5000 [==============================] - 2s 441us/step - loss: 672.3518
Epoch 9/25
5000/5000 [==============================] - 2s 433us/step - loss: 611.2807
Epoch 10/25
5000/5000 [==============================] - 2s 435us/step - loss: 554.2607
Epoch 11/25
5000/5000 [==============================] - 2s 441us/step - loss: 501.7422
Epoch 12/25
5000/5000 [==============================] - 2s 443us/step - loss: 453.8701
Epoch 13/25
5000/5000 [==============================] - 2s 439us/step - loss: 410.1427
Epoch 14/25
5000/5000 [==============================] - 2s 434us/step - loss: 370.2551
Epoch 15/25
5000/5000 [==============================] - 2s 445us/step - loss: 334.1062
Epoch 16/25
5000/5000 [==============================] - 2s 461us/step - loss: 301.5923
Epoch 17/25
5000/5000 [==============================] - 2s 441us/step - loss: 272.4631
Epoch 18/25
5000/5000 [==============================] - 2s 447us/step - loss: 246.9667
Epoch 19/25
5000/5000 [==============================] - 2s 445us/step - loss: 225.1783
Epoch 20/25
5000/5000 [==============================] - 2s 449us/step - loss: 206.6166
Epoch 21/25
5000/5000 [==============================] - 2s 449us/step - loss: 190.8524
Epoch 22/25
5000/5000 [==============================] - 2s 446us/step - loss: 177.7150
Epoch 23/25
5000/5000 [==============================] - 2s 450us/step - loss: 166.9464
Epoch 24/25
5000/5000 [==============================] - 2s 449us/step - loss: 158.0545
Epoch 25/25
5000/5000 [==============================] - 2s 446us/step - loss: 150.5878
mse lin simec: 12.67361
mse lin simec pred: 12.53565
Epoch 1/25
5000/5000 [==============================] - 4s 812us/step - loss: 0.0695
Epoch 2/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0467
Epoch 3/25
5000/5000 [==============================] - 2s 474us/step - loss: 0.0374
Epoch 4/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0313
Epoch 5/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0290
Epoch 6/25
5000/5000 [==============================] - 2s 474us/step - loss: 0.0279
Epoch 7/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0272
Epoch 8/25
5000/5000 [==============================] - 2s 474us/step - loss: 0.0265
Epoch 9/25
5000/5000 [==============================] - 2s 476us/step - loss: 0.0258
Epoch 10/25
5000/5000 [==============================] - 2s 476us/step - loss: 0.0251
Epoch 11/25
5000/5000 [==============================] - 2s 478us/step - loss: 0.0242
Epoch 12/25
5000/5000 [==============================] - 2s 474us/step - loss: 0.0231
Epoch 13/25
5000/5000 [==============================] - 2s 475us/step - loss: 0.0221
Epoch 14/25
5000/5000 [==============================] - 2s 478us/step - loss: 0.0213
Epoch 15/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0208
Epoch 16/25
5000/5000 [==============================] - 2s 474us/step - loss: 0.0204
Epoch 17/25
5000/5000 [==============================] - 2s 473us/step - loss: 0.0202
Epoch 18/25
5000/5000 [==============================] - 2s 474us/step - loss: 0.0201
Epoch 19/25
5000/5000 [==============================] - 2s 473us/step - loss: 0.0201
Epoch 20/25
5000/5000 [==============================] - 2s 474us/step - loss: 0.0201
Epoch 21/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0200
Epoch 22/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0200
Epoch 23/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0200
Epoch 24/25
5000/5000 [==============================] - 2s 476us/step - loss: 0.0200
Epoch 25/25
5000/5000 [==============================] - 2s 479us/step - loss: 0.0199
mse class simec: 0.01818
mse class simec pred: 0.01775
noise dimensions: 4704
mse lin kpca: 11.18668
Epoch 1/25
5000/5000 [==============================] - 4s 788us/step - loss: 991.1993
Epoch 2/25
5000/5000 [==============================] - 2s 456us/step - loss: 960.9286
Epoch 3/25
5000/5000 [==============================] - 2s 453us/step - loss: 931.5190
Epoch 4/25
5000/5000 [==============================] - 2s 451us/step - loss: 895.3144
Epoch 5/25
5000/5000 [==============================] - 2s 455us/step - loss: 850.4417
Epoch 6/25
5000/5000 [==============================] - 2s 457us/step - loss: 796.1273
Epoch 7/25
5000/5000 [==============================] - 2s 450us/step - loss: 735.5443
Epoch 8/25
5000/5000 [==============================] - 2s 454us/step - loss: 673.0272
Epoch 9/25
5000/5000 [==============================] - 2s 455us/step - loss: 611.6190
Epoch 10/25
5000/5000 [==============================] - 2s 455us/step - loss: 553.1399
Epoch 11/25
5000/5000 [==============================] - 2s 451us/step - loss: 498.7143
Epoch 12/25
5000/5000 [==============================] - 2s 459us/step - loss: 449.1968
Epoch 13/25
5000/5000 [==============================] - 2s 457us/step - loss: 404.9699
Epoch 14/25
5000/5000 [==============================] - 2s 454us/step - loss: 365.7057
Epoch 15/25
5000/5000 [==============================] - 2s 452us/step - loss: 330.6278
Epoch 16/25
5000/5000 [==============================] - 2s 453us/step - loss: 299.0293
Epoch 17/25
5000/5000 [==============================] - 2s 453us/step - loss: 270.7030
Epoch 18/25
5000/5000 [==============================] - 2s 451us/step - loss: 245.7374
Epoch 19/25
5000/5000 [==============================] - 2s 454us/step - loss: 224.0441
Epoch 20/25
5000/5000 [==============================] - 2s 454us/step - loss: 205.1188
Epoch 21/25
5000/5000 [==============================] - 2s 455us/step - loss: 188.4878
Epoch 22/25
5000/5000 [==============================] - 2s 456us/step - loss: 174.0295
Epoch 23/25
5000/5000 [==============================] - 2s 454us/step - loss: 161.6928
Epoch 24/25
5000/5000 [==============================] - 2s 455us/step - loss: 151.5282
Epoch 25/25
5000/5000 [==============================] - 2s 457us/step - loss: 143.5115
mse lin simec: 11.93778
mse lin simec pred: 12.00615
Epoch 1/25
5000/5000 [==============================] - 4s 816us/step - loss: 0.0719
Epoch 2/25
5000/5000 [==============================] - 2s 480us/step - loss: 0.0476
Epoch 3/25
5000/5000 [==============================] - 2s 481us/step - loss: 0.0369
Epoch 4/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0305
Epoch 5/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0287
Epoch 6/25
5000/5000 [==============================] - 2s 484us/step - loss: 0.0278
Epoch 7/25
5000/5000 [==============================] - 2s 478us/step - loss: 0.0271
Epoch 8/25
5000/5000 [==============================] - 2s 487us/step - loss: 0.0265
Epoch 9/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0259
Epoch 10/25
5000/5000 [==============================] - 2s 481us/step - loss: 0.0251
Epoch 11/25
5000/5000 [==============================] - 2s 483us/step - loss: 0.0241
Epoch 12/25
5000/5000 [==============================] - 2s 483us/step - loss: 0.0231
Epoch 13/25
5000/5000 [==============================] - 2s 480us/step - loss: 0.0221
Epoch 14/25
5000/5000 [==============================] - 2s 477us/step - loss: 0.0213
Epoch 15/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0209
Epoch 16/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0206
Epoch 17/25
5000/5000 [==============================] - 2s 486us/step - loss: 0.0205
Epoch 18/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0204
Epoch 19/25
5000/5000 [==============================] - 2s 483us/step - loss: 0.0203
Epoch 20/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0203
Epoch 21/25
5000/5000 [==============================] - 2s 484us/step - loss: 0.0202
Epoch 22/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0201
Epoch 23/25
5000/5000 [==============================] - 2s 487us/step - loss: 0.0201
Epoch 24/25
5000/5000 [==============================] - 2s 486us/step - loss: 0.0201
Epoch 25/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0200
mse class simec: 0.01817
mse class simec pred: 0.01779
noise dimensions: 6272
mse lin kpca: 11.25246
Epoch 1/25
5000/5000 [==============================] - 4s 801us/step - loss: 991.3082
Epoch 2/25
5000/5000 [==============================] - 2s 463us/step - loss: 959.9177
Epoch 3/25
5000/5000 [==============================] - 2s 459us/step - loss: 929.0864
Epoch 4/25
5000/5000 [==============================] - 2s 460us/step - loss: 897.0950
Epoch 5/25
5000/5000 [==============================] - 2s 459us/step - loss: 852.7554
Epoch 6/25
5000/5000 [==============================] - 2s 460us/step - loss: 797.0779
Epoch 7/25
5000/5000 [==============================] - 2s 459us/step - loss: 734.9298
Epoch 8/25
5000/5000 [==============================] - 2s 459us/step - loss: 671.5316
Epoch 9/25
5000/5000 [==============================] - 2s 458us/step - loss: 611.0258
Epoch 10/25
5000/5000 [==============================] - 2s 460us/step - loss: 555.8563
Epoch 11/25
5000/5000 [==============================] - 2s 457us/step - loss: 505.7877
Epoch 12/25
5000/5000 [==============================] - 2s 460us/step - loss: 459.5909
Epoch 13/25
5000/5000 [==============================] - 2s 460us/step - loss: 416.5346
Epoch 14/25
5000/5000 [==============================] - 2s 461us/step - loss: 376.6941
Epoch 15/25
5000/5000 [==============================] - 2s 458us/step - loss: 340.4046
Epoch 16/25
5000/5000 [==============================] - 2s 460us/step - loss: 307.6889
Epoch 17/25
5000/5000 [==============================] - 2s 457us/step - loss: 278.3179
Epoch 18/25
5000/5000 [==============================] - 2s 457us/step - loss: 252.0088
Epoch 19/25
5000/5000 [==============================] - 2s 461us/step - loss: 228.8351
Epoch 20/25
5000/5000 [==============================] - 2s 459us/step - loss: 208.5897
Epoch 21/25
5000/5000 [==============================] - 2s 456us/step - loss: 190.9148
Epoch 22/25
5000/5000 [==============================] - 2s 459us/step - loss: 175.7061
Epoch 23/25
5000/5000 [==============================] - 2s 457us/step - loss: 162.9449
Epoch 24/25
5000/5000 [==============================] - 2s 461us/step - loss: 152.5930
Epoch 25/25
5000/5000 [==============================] - 2s 458us/step - loss: 144.5687
mse lin simec: 12.19764
mse lin simec pred: 12.16781
Epoch 1/25
5000/5000 [==============================] - 4s 840us/step - loss: 0.0729
Epoch 2/25
5000/5000 [==============================] - 2s 489us/step - loss: 0.0447
Epoch 3/25
5000/5000 [==============================] - 2s 483us/step - loss: 0.0325
Epoch 4/25
5000/5000 [==============================] - 2s 480us/step - loss: 0.0290
Epoch 5/25
5000/5000 [==============================] - 2s 480us/step - loss: 0.0278
Epoch 6/25
5000/5000 [==============================] - 2s 486us/step - loss: 0.0271
Epoch 7/25
5000/5000 [==============================] - 2s 487us/step - loss: 0.0265
Epoch 8/25
5000/5000 [==============================] - 2s 487us/step - loss: 0.0258
Epoch 9/25
5000/5000 [==============================] - 2s 489us/step - loss: 0.0250
Epoch 10/25
5000/5000 [==============================] - 2s 490us/step - loss: 0.0241
Epoch 11/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0231
Epoch 12/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0221
Epoch 13/25
5000/5000 [==============================] - 2s 483us/step - loss: 0.0212
Epoch 14/25
5000/5000 [==============================] - 2s 486us/step - loss: 0.0207
Epoch 15/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0206
Epoch 16/25
5000/5000 [==============================] - 2s 483us/step - loss: 0.0205
Epoch 17/25
5000/5000 [==============================] - 2s 490us/step - loss: 0.0204
Epoch 18/25
5000/5000 [==============================] - 2s 484us/step - loss: 0.0204
Epoch 19/25
5000/5000 [==============================] - 2s 483us/step - loss: 0.0203
Epoch 20/25
5000/5000 [==============================] - 2s 491us/step - loss: 0.0203
Epoch 21/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0203
Epoch 22/25
5000/5000 [==============================] - 2s 489us/step - loss: 0.0202
Epoch 23/25
5000/5000 [==============================] - 2s 490us/step - loss: 0.0202
Epoch 24/25
5000/5000 [==============================] - 2s 481us/step - loss: 0.0202
Epoch 25/25
5000/5000 [==============================] - 2s 488us/step - loss: 0.0201
mse class simec: 0.01835
mse class simec pred: 0.01792
noise dimensions: 7840
mse lin kpca: 11.32829
Epoch 1/25
5000/5000 [==============================] - 4s 815us/step - loss: 991.4825
Epoch 2/25
5000/5000 [==============================] - 2s 468us/step - loss: 959.9860
Epoch 3/25
5000/5000 [==============================] - 2s 468us/step - loss: 929.7749
Epoch 4/25
5000/5000 [==============================] - 2s 468us/step - loss: 897.5714
Epoch 5/25
5000/5000 [==============================] - 2s 470us/step - loss: 853.6245
Epoch 6/25
5000/5000 [==============================] - 2s 469us/step - loss: 799.9002
Epoch 7/25
5000/5000 [==============================] - 2s 468us/step - loss: 739.7231
Epoch 8/25
5000/5000 [==============================] - 2s 465us/step - loss: 676.6750
Epoch 9/25
5000/5000 [==============================] - 2s 462us/step - loss: 615.5018
Epoch 10/25
5000/5000 [==============================] - 2s 463us/step - loss: 558.6427
Epoch 11/25
5000/5000 [==============================] - 2s 466us/step - loss: 506.4658
Epoch 12/25
5000/5000 [==============================] - 2s 464us/step - loss: 458.1732
Epoch 13/25
5000/5000 [==============================] - 2s 465us/step - loss: 413.5865
Epoch 14/25
5000/5000 [==============================] - 2s 468us/step - loss: 372.6300
Epoch 15/25
5000/5000 [==============================] - 2s 466us/step - loss: 335.6340
Epoch 16/25
5000/5000 [==============================] - 2s 466us/step - loss: 302.3815
Epoch 17/25
5000/5000 [==============================] - 2s 465us/step - loss: 272.1885
Epoch 18/25
5000/5000 [==============================] - 2s 465us/step - loss: 245.0753
Epoch 19/25
5000/5000 [==============================] - 2s 468us/step - loss: 221.3090
Epoch 20/25
5000/5000 [==============================] - 2s 468us/step - loss: 201.0664
Epoch 21/25
5000/5000 [==============================] - 2s 469us/step - loss: 184.3722
Epoch 22/25
5000/5000 [==============================] - 2s 469us/step - loss: 170.9814
Epoch 23/25
5000/5000 [==============================] - 2s 466us/step - loss: 160.5446
Epoch 24/25
5000/5000 [==============================] - 2s 467us/step - loss: 152.5601
Epoch 25/25
5000/5000 [==============================] - 2s 468us/step - loss: 146.4541
mse lin simec: 12.94194
mse lin simec pred: 12.63251
Epoch 1/25
5000/5000 [==============================] - 4s 852us/step - loss: 0.0723
Epoch 2/25
5000/5000 [==============================] - 2s 495us/step - loss: 0.0438
Epoch 3/25
5000/5000 [==============================] - 2s 494us/step - loss: 0.0312
Epoch 4/25
5000/5000 [==============================] - 2s 493us/step - loss: 0.0285
Epoch 5/25
5000/5000 [==============================] - 2s 491us/step - loss: 0.0275
Epoch 6/25
5000/5000 [==============================] - 2s 491us/step - loss: 0.0270
Epoch 7/25
5000/5000 [==============================] - 2s 491us/step - loss: 0.0266
Epoch 8/25
5000/5000 [==============================] - 2s 495us/step - loss: 0.0261
Epoch 9/25
5000/5000 [==============================] - 2s 495us/step - loss: 0.0255
Epoch 10/25
5000/5000 [==============================] - 2s 491us/step - loss: 0.0249
Epoch 11/25
5000/5000 [==============================] - 2s 496us/step - loss: 0.0242
Epoch 12/25
5000/5000 [==============================] - 2s 493us/step - loss: 0.0235
Epoch 13/25
5000/5000 [==============================] - 2s 495us/step - loss: 0.0228
Epoch 14/25
5000/5000 [==============================] - 2s 490us/step - loss: 0.0220
Epoch 15/25
5000/5000 [==============================] - 2s 486us/step - loss: 0.0213
Epoch 16/25
5000/5000 [==============================] - 2s 488us/step - loss: 0.0209
Epoch 17/25
5000/5000 [==============================] - 2s 493us/step - loss: 0.0207
Epoch 18/25
5000/5000 [==============================] - 2s 492us/step - loss: 0.0205
Epoch 19/25
5000/5000 [==============================] - 2s 493us/step - loss: 0.0204
Epoch 20/25
5000/5000 [==============================] - 2s 494us/step - loss: 0.0204
Epoch 21/25
5000/5000 [==============================] - 2s 492us/step - loss: 0.0203
Epoch 22/25
5000/5000 [==============================] - 2s 494us/step - loss: 0.0203
Epoch 23/25
5000/5000 [==============================] - 2s 490us/step - loss: 0.0203
Epoch 24/25
5000/5000 [==============================] - 2s 493us/step - loss: 0.0203
Epoch 25/25
5000/5000 [==============================] - 2s 497us/step - loss: 0.0203
mse class simec: 0.01847
mse class simec pred: 0.01802
mse_lin_kpca= [11.048889013360812, 11.064638760599191, 11.083612724298947, 11.13001880000037, 11.186681309879853, 11.252460875260446, 11.328288445721379]
mse_lin_simec= [11.072531680109639, 11.325954150692651, 11.635435636712177, 12.673614111116194, 11.937778366115552, 12.197635194151857, 12.941944756578208]
mse_class_simec= [0.022851555115145212, 0.019234524517200872, 0.018881540172807586, 0.018179413419462415, 0.018166466094256963, 0.018346011505090472, 0.018474664804292891]
mse_lin_simec_pred= [11.061521754819577, 11.17769614278208, 11.338745399957938, 12.535650505018769, 12.006145945189635, 12.167805827514425, 12.632510894522808]
mse_class_simec_pred= [0.020266552970292527, 0.01830834570110337, 0.018100770134566586, 0.017745864819502435, 0.017793522955823468, 0.0179214970145029, 0.018019924517878736]






    Out[5]:





Text(0,0.5,u'Mean Squared Error of $\\hat{S}$')

Add additional input dimensions with correlated noise



In [6]:

    
np.random.seed(15)
noise_levels = [0, X.shape[1], 2*X.shape[1], 4*X.shape[1], 6*X.shape[1], 8*X.shape[1], 10*X.shape[1]]
mse_lin_kpca = []
mse_lin_simec = []
mse_class_simec = []
mse_lin_simec_pred = []
mse_class_simec_pred = []
# generate random covariance matrix
cov = np.random.randn(max(noise_levels), max(noise_levels))
cov = 0.33*(cov + cov.T + np.eye(max(noise_levels)))
# make sure it's positive semidefinite
D, V = np.linalg.eig(cov)
D, V = D[np.argsort(D)[::-1]], V[:, np.argsort(D)[::-1]]
cov_embed = np.dot(V, np.diag(np.sqrt(np.abs(D))))
cov = np.dot(cov_embed[:,:100], cov_embed[:,:100].T)
# generate correlated noise data
noise_train = np.random.multivariate_normal(np.zeros(max(noise_levels)), cov, X.shape[0])
noise_train = x_std*noise_train/np.std(noise_train)
for noise_level in noise_levels:
    print("noise dimensions: %i" % noise_level)
    # add random noise as additional feature dimensions
    X_noisy = np.hstack((X, noise_train[:,:noise_level]))
    # lin kpca
    kpca = KernelPCA(n_components=10, kernel='linear')
    mse_lin_kpca.append(check_similarity_match(kpca.fit_transform(X_noisy), K_lin)[0])
    print("mse lin kpca: %.5f" % mse_lin_kpca[-1])
    # lin simec
    simec = SimilarityEncoder(X_noisy.shape[1], 10, n_targets, s_ll_reg=10., S_ll=K_lin[:n_targets,:n_targets],
                              l2_reg_emb=0.001, l2_reg_out=0.00001, opt=keras.optimizers.Adamax(lr=0.005))
    simec.fit(X_noisy, K_lin[:, :n_targets])
    mse_lin_simec.append(check_similarity_match(simec.transform(X_noisy), K_lin)[0])
    print("mse lin simec: %.5f" % mse_lin_simec[-1])
    mse_lin_simec_pred.append(check_similarity_match(simec.predict(X_noisy), K_lin, X_embed_is_S_approx=True)[0])
    print("mse lin simec pred: %.5f" % mse_lin_simec_pred[-1])
    # class simec
    simec = SimilarityEncoder(X_noisy.shape[1], 7, n_targets, hidden_layers=[(25, 'tanh'), (25, 'tanh')],
                              s_ll_reg=0.1, S_ll=S[:n_targets,:n_targets], orth_reg=0.00001, l2_reg=0.,
                              l2_reg_emb=0.00001, l2_reg_out=0.0000001, opt=keras.optimizers.Adamax(lr=0.01))
    simec.fit(X_noisy, S[:, :n_targets])
    mse_class_simec.append(check_similarity_match(simec.transform(X_noisy), S)[0])
    print("mse class simec: %.5f" % mse_class_simec[-1])
    mse_class_simec_pred.append(check_similarity_match(simec.predict(X_noisy), S, X_embed_is_S_approx=True)[0])
    print("mse class simec pred: %.5f" % mse_class_simec_pred[-1])
keras.backend.clear_session()
print("mse_lin_kpca=", mse_lin_kpca)
print("mse_lin_simec=", mse_lin_simec)
print("mse_class_simec=", mse_class_simec)
print("mse_lin_simec_pred=", mse_lin_simec_pred)
print("mse_class_simec_pred=", mse_class_simec_pred)
colors = get_colors(10)
plt.figure();
plt.plot([0, noise_levels[-1]], [mse_lk_ed, mse_lk_ed], '--', c=colors[4], linewidth=0.5, label='10D lin kPCA (no noise)');
plt.plot(noise_levels, mse_lin_kpca, '-o', markersize=3, c=colors[4], label='10D lin kPCA');
plt.plot(noise_levels, mse_lin_simec, '-o', markersize=3, c=colors[6], label='10D lin SimEc ($YY^{\\top}$)');
plt.plot(noise_levels, mse_lin_simec_pred, '--', markersize=3, c=colors[6], label='10D lin SimEc ($YW_l$)');
plt.legend(loc=0);
plt.title('Added dimensions with correlated noise (linear kernel)');
plt.xticks(noise_levels, [0, '$D$', '$2\\times D$', '$4\\times D$', '$6\\times D$', '$8\\times D$', '$10\\times D$']);
plt.xlabel('Additional noise dimensions (where $D = 784$)')
plt.ylabel('Mean Squared Error of $\hat{S}$')
plt.figure();
plt.plot([0, noise_levels[-1]], [mse_class_ed, mse_class_ed], '--', c=colors[8], linewidth=0.5, label='7D class ED (no noise)');
plt.plot(noise_levels, mse_class_simec, '-o', markersize=3, c=colors[8], label='7D class SimEc ($YY^{\\top}$)');
plt.plot(noise_levels, mse_class_simec_pred, '--', markersize=3, c=colors[8], label='7D class SimEc ($YW_l$)');
plt.legend(loc=0);
plt.title('Added dimensions with correlated noise (class sim)');
plt.xticks(noise_levels, [0, '$D$', '$2\\times D$', '$4\\times D$', '$6\\times D$', '$8\\times D$', '$10\\times D$']);
plt.xlabel('Additional noise dimensions (where $D = 784$)')
plt.ylabel('Mean Squared Error of $\hat{S}$');









    



noise dimensions: 0
mse lin kpca: 11.04889
Epoch 1/25
5000/5000 [==============================] - 4s 703us/step - loss: 928.9582
Epoch 2/25
5000/5000 [==============================] - 2s 438us/step - loss: 680.2921
Epoch 3/25
5000/5000 [==============================] - 2s 438us/step - loss: 430.4882
Epoch 4/25
5000/5000 [==============================] - 2s 437us/step - loss: 274.2560
Epoch 5/25
5000/5000 [==============================] - 2s 439us/step - loss: 191.4925
Epoch 6/25
5000/5000 [==============================] - 2s 439us/step - loss: 155.5927
Epoch 7/25
5000/5000 [==============================] - 2s 444us/step - loss: 141.2990
Epoch 8/25
5000/5000 [==============================] - 2s 439us/step - loss: 134.9620
Epoch 9/25
5000/5000 [==============================] - 2s 441us/step - loss: 131.7087
Epoch 10/25
5000/5000 [==============================] - 2s 441us/step - loss: 129.4775
Epoch 11/25
5000/5000 [==============================] - 2s 442us/step - loss: 127.5119
Epoch 12/25
5000/5000 [==============================] - 2s 441us/step - loss: 125.7327
Epoch 13/25
5000/5000 [==============================] - 2s 441us/step - loss: 124.3406
Epoch 14/25
5000/5000 [==============================] - 2s 444us/step - loss: 123.4805
Epoch 15/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.0673
Epoch 16/25
5000/5000 [==============================] - 2s 441us/step - loss: 122.9171
Epoch 17/25
5000/5000 [==============================] - 2s 444us/step - loss: 122.8709
Epoch 18/25
5000/5000 [==============================] - 2s 442us/step - loss: 122.8589
Epoch 19/25
5000/5000 [==============================] - 2s 439us/step - loss: 122.8515
Epoch 20/25
5000/5000 [==============================] - 2s 443us/step - loss: 122.8538
Epoch 21/25
5000/5000 [==============================] - 2s 439us/step - loss: 122.8538
Epoch 22/25
5000/5000 [==============================] - 2s 441us/step - loss: 122.8456
Epoch 23/25
5000/5000 [==============================] - 2s 442us/step - loss: 122.8536
Epoch 24/25
5000/5000 [==============================] - 2s 442us/step - loss: 122.8582
Epoch 25/25
5000/5000 [==============================] - 2s 442us/step - loss: 122.8559
mse lin simec: 11.06733
mse lin simec pred: 11.05838
Epoch 1/25
5000/5000 [==============================] - 4s 734us/step - loss: 0.0587
Epoch 2/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0434
Epoch 3/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0393
Epoch 4/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0367
Epoch 5/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0348
Epoch 6/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0330
Epoch 7/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0311
Epoch 8/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0295
Epoch 9/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0283
Epoch 10/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0273
Epoch 11/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0266
Epoch 12/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0261
Epoch 13/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0258
Epoch 14/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0256
Epoch 15/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0251
Epoch 16/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0249
Epoch 17/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0246
Epoch 18/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0243
Epoch 19/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0241
Epoch 20/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0238
Epoch 21/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0237
Epoch 22/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0235
Epoch 23/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0233
Epoch 24/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0232
Epoch 25/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0230
mse class simec: 0.02298
mse class simec pred: 0.02043
noise dimensions: 784
mse lin kpca: 11.37393
Epoch 1/25
5000/5000 [==============================] - 4s 706us/step - loss: 930.6201
Epoch 2/25
5000/5000 [==============================] - 2s 440us/step - loss: 687.3936
Epoch 3/25
5000/5000 [==============================] - 2s 438us/step - loss: 432.7382
Epoch 4/25
5000/5000 [==============================] - 2s 444us/step - loss: 285.6831
Epoch 5/25
5000/5000 [==============================] - 2s 443us/step - loss: 200.2104
Epoch 6/25
5000/5000 [==============================] - 2s 441us/step - loss: 157.5510
Epoch 7/25
5000/5000 [==============================] - 2s 439us/step - loss: 140.5998
Epoch 8/25
5000/5000 [==============================] - 2s 439us/step - loss: 133.4970
Epoch 9/25
5000/5000 [==============================] - 2s 440us/step - loss: 129.9888
Epoch 10/25
5000/5000 [==============================] - 2s 441us/step - loss: 127.3448
Epoch 11/25
5000/5000 [==============================] - 2s 443us/step - loss: 125.2619
Epoch 12/25
5000/5000 [==============================] - 2s 439us/step - loss: 124.0346
Epoch 13/25
5000/5000 [==============================] - 2s 438us/step - loss: 123.3876
Epoch 14/25
5000/5000 [==============================] - 2s 439us/step - loss: 123.0522
Epoch 15/25
5000/5000 [==============================] - 2s 436us/step - loss: 123.0236
Epoch 16/25
5000/5000 [==============================] - 2s 437us/step - loss: 122.9336
Epoch 17/25
5000/5000 [==============================] - 2s 437us/step - loss: 122.9577
Epoch 18/25
5000/5000 [==============================] - 2s 437us/step - loss: 122.9596
Epoch 19/25
5000/5000 [==============================] - 2s 437us/step - loss: 122.9343
Epoch 20/25
5000/5000 [==============================] - 2s 442us/step - loss: 122.9738
Epoch 21/25
5000/5000 [==============================] - 2s 440us/step - loss: 122.9259
Epoch 22/25
5000/5000 [==============================] - 2s 440us/step - loss: 122.9771
Epoch 23/25
5000/5000 [==============================] - 2s 438us/step - loss: 122.9519
Epoch 24/25
5000/5000 [==============================] - 2s 437us/step - loss: 122.9580
Epoch 25/25
5000/5000 [==============================] - 2s 438us/step - loss: 122.9141
mse lin simec: 11.17193
mse lin simec pred: 11.11059
Epoch 1/25
5000/5000 [==============================] - 4s 740us/step - loss: 0.0595
Epoch 2/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0426
Epoch 3/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0385
Epoch 4/25
5000/5000 [==============================] - 2s 460us/step - loss: 0.0361
Epoch 5/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0341
Epoch 6/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0323
Epoch 7/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0307
Epoch 8/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0290
Epoch 9/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0274
Epoch 10/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0262
Epoch 11/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0255
Epoch 12/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0250
Epoch 13/25
5000/5000 [==============================] - 2s 468us/step - loss: 0.0247
Epoch 14/25
5000/5000 [==============================] - 2s 462us/step - loss: 0.0244
Epoch 15/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0241
Epoch 16/25
5000/5000 [==============================] - 2s 464us/step - loss: 0.0239
Epoch 17/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0236
Epoch 18/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0234
Epoch 19/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0233
Epoch 20/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0231
Epoch 21/25
5000/5000 [==============================] - 2s 463us/step - loss: 0.0230
Epoch 22/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0229
Epoch 23/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0228
Epoch 24/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0227
Epoch 25/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0227
mse class simec: 0.02274
mse class simec pred: 0.02020
noise dimensions: 1568
mse lin kpca: 24.90956
Epoch 1/25
5000/5000 [==============================] - 4s 714us/step - loss: 928.8736
Epoch 2/25
5000/5000 [==============================] - 2s 439us/step - loss: 682.1742
Epoch 3/25
5000/5000 [==============================] - 2s 440us/step - loss: 431.8938
Epoch 4/25
5000/5000 [==============================] - 2s 439us/step - loss: 283.4505
Epoch 5/25
5000/5000 [==============================] - 2s 443us/step - loss: 200.5378
Epoch 6/25
5000/5000 [==============================] - 2s 445us/step - loss: 159.3989
Epoch 7/25
5000/5000 [==============================] - 2s 441us/step - loss: 142.6026
Epoch 8/25
5000/5000 [==============================] - 2s 442us/step - loss: 136.8815
Epoch 9/25
5000/5000 [==============================] - 2s 442us/step - loss: 134.5917
Epoch 10/25
5000/5000 [==============================] - 2s 441us/step - loss: 133.1837
Epoch 11/25
5000/5000 [==============================] - 2s 445us/step - loss: 132.1790
Epoch 12/25
5000/5000 [==============================] - 2s 446us/step - loss: 130.6530
Epoch 13/25
5000/5000 [==============================] - 2s 444us/step - loss: 127.8907
Epoch 14/25
5000/5000 [==============================] - 2s 446us/step - loss: 125.1286
Epoch 15/25
5000/5000 [==============================] - 2s 445us/step - loss: 123.6051
Epoch 16/25
5000/5000 [==============================] - 2s 444us/step - loss: 123.1930
Epoch 17/25
5000/5000 [==============================] - 2s 444us/step - loss: 123.1086
Epoch 18/25
5000/5000 [==============================] - 2s 440us/step - loss: 123.1273
Epoch 19/25
5000/5000 [==============================] - 2s 444us/step - loss: 123.1345
Epoch 20/25
5000/5000 [==============================] - 2s 447us/step - loss: 123.1145
Epoch 21/25
5000/5000 [==============================] - 2s 440us/step - loss: 123.1552
Epoch 22/25
5000/5000 [==============================] - 2s 443us/step - loss: 123.1314
Epoch 23/25
5000/5000 [==============================] - 2s 444us/step - loss: 123.1094
Epoch 24/25
5000/5000 [==============================] - 2s 445us/step - loss: 123.0914
Epoch 25/25
5000/5000 [==============================] - 2s 444us/step - loss: 123.1256
mse lin simec: 11.48115
mse lin simec pred: 11.24044
Epoch 1/25
5000/5000 [==============================] - 4s 742us/step - loss: 0.0663
Epoch 2/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0461
Epoch 3/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0408
Epoch 4/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0379
Epoch 5/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0355
Epoch 6/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0335
Epoch 7/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0317
Epoch 8/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0299
Epoch 9/25
5000/5000 [==============================] - 2s 461us/step - loss: 0.0286
Epoch 10/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0274
Epoch 11/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0266
Epoch 12/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0260
Epoch 13/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0257
Epoch 14/25
5000/5000 [==============================] - 2s 475us/step - loss: 0.0253
Epoch 15/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0249
Epoch 16/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0246
Epoch 17/25
5000/5000 [==============================] - 2s 467us/step - loss: 0.0245
Epoch 18/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0242
Epoch 19/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0240
Epoch 20/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0237
Epoch 21/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0236
Epoch 22/25
5000/5000 [==============================] - 2s 466us/step - loss: 0.0235
Epoch 23/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0233
Epoch 24/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0232
Epoch 25/25
5000/5000 [==============================] - 2s 465us/step - loss: 0.0231
mse class simec: 0.02316
mse class simec pred: 0.02049
noise dimensions: 3136
mse lin kpca: 89.55807
Epoch 1/25
5000/5000 [==============================] - 4s 729us/step - loss: 933.7599
Epoch 2/25
5000/5000 [==============================] - 2s 448us/step - loss: 691.9400
Epoch 3/25
5000/5000 [==============================] - 2s 447us/step - loss: 444.2383
Epoch 4/25
5000/5000 [==============================] - 2s 445us/step - loss: 290.3510
Epoch 5/25
5000/5000 [==============================] - 2s 449us/step - loss: 203.4922
Epoch 6/25
5000/5000 [==============================] - 2s 450us/step - loss: 161.1623
Epoch 7/25
5000/5000 [==============================] - 2s 449us/step - loss: 141.3698
Epoch 8/25
5000/5000 [==============================] - 2s 453us/step - loss: 131.5618
Epoch 9/25
5000/5000 [==============================] - 2s 450us/step - loss: 126.8156
Epoch 10/25
5000/5000 [==============================] - 2s 450us/step - loss: 125.1337
Epoch 11/25
5000/5000 [==============================] - 2s 444us/step - loss: 123.9335
Epoch 12/25
5000/5000 [==============================] - 2s 443us/step - loss: 123.6197
Epoch 13/25
5000/5000 [==============================] - 2s 449us/step - loss: 123.8282
Epoch 14/25
5000/5000 [==============================] - 2s 455us/step - loss: 123.5506
Epoch 15/25
5000/5000 [==============================] - 2s 450us/step - loss: 123.8669
Epoch 16/25
5000/5000 [==============================] - 2s 448us/step - loss: 123.6589
Epoch 17/25
5000/5000 [==============================] - 2s 450us/step - loss: 123.7003
Epoch 18/25
5000/5000 [==============================] - 2s 447us/step - loss: 123.4432
Epoch 19/25
5000/5000 [==============================] - 2s 451us/step - loss: 123.6607
Epoch 20/25
5000/5000 [==============================] - 2s 451us/step - loss: 123.5552
Epoch 21/25
5000/5000 [==============================] - 2s 447us/step - loss: 123.6408
Epoch 22/25
5000/5000 [==============================] - 2s 449us/step - loss: 123.5411
Epoch 23/25
5000/5000 [==============================] - 2s 451us/step - loss: 123.6993
Epoch 24/25
5000/5000 [==============================] - 2s 452us/step - loss: 123.5648
Epoch 25/25
5000/5000 [==============================] - 2s 442us/step - loss: 123.6012
mse lin simec: 12.35582
mse lin simec pred: 11.71772
Epoch 1/25
5000/5000 [==============================] - 4s 764us/step - loss: 0.0739
Epoch 2/25
5000/5000 [==============================] - 2s 475us/step - loss: 0.0540
Epoch 3/25
5000/5000 [==============================] - 2s 474us/step - loss: 0.0462
Epoch 4/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0416
Epoch 5/25
5000/5000 [==============================] - 2s 475us/step - loss: 0.0390
Epoch 6/25
5000/5000 [==============================] - 2s 477us/step - loss: 0.0364
Epoch 7/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0346
Epoch 8/25
5000/5000 [==============================] - 2s 478us/step - loss: 0.0327
Epoch 9/25
5000/5000 [==============================] - 2s 477us/step - loss: 0.0312
Epoch 10/25
5000/5000 [==============================] - 2s 469us/step - loss: 0.0297
Epoch 11/25
5000/5000 [==============================] - 2s 475us/step - loss: 0.0283
Epoch 12/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0277
Epoch 13/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0270
Epoch 14/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0267
Epoch 15/25
5000/5000 [==============================] - 2s 471us/step - loss: 0.0263
Epoch 16/25
5000/5000 [==============================] - 2s 477us/step - loss: 0.0260
Epoch 17/25
5000/5000 [==============================] - 2s 472us/step - loss: 0.0259
Epoch 18/25
5000/5000 [==============================] - 2s 470us/step - loss: 0.0256
Epoch 19/25
5000/5000 [==============================] - 2s 473us/step - loss: 0.0252
Epoch 20/25
5000/5000 [==============================] - 2s 475us/step - loss: 0.0249
Epoch 21/25
5000/5000 [==============================] - 2s 477us/step - loss: 0.0247
Epoch 22/25
5000/5000 [==============================] - 2s 476us/step - loss: 0.0247
Epoch 23/25
5000/5000 [==============================] - 2s 477us/step - loss: 0.0244
Epoch 24/25
5000/5000 [==============================] - 2s 475us/step - loss: 0.0242
Epoch 25/25
5000/5000 [==============================] - 2s 477us/step - loss: 0.0241
mse class simec: 0.02483
mse class simec pred: 0.02145
noise dimensions: 4704
mse lin kpca: 199.01193
Epoch 1/25
5000/5000 [==============================] - 4s 746us/step - loss: 937.9327
Epoch 2/25
5000/5000 [==============================] - 2s 456us/step - loss: 711.7534
Epoch 3/25
5000/5000 [==============================] - 2s 450us/step - loss: 450.5065
Epoch 4/25
5000/5000 [==============================] - 2s 453us/step - loss: 299.8572
Epoch 5/25
5000/5000 [==============================] - 2s 455us/step - loss: 216.6130
Epoch 6/25
5000/5000 [==============================] - 2s 454us/step - loss: 167.0011
Epoch 7/25
5000/5000 [==============================] - 2s 452us/step - loss: 139.1705
Epoch 8/25
5000/5000 [==============================] - 2s 453us/step - loss: 130.2044
Epoch 9/25
5000/5000 [==============================] - 2s 453us/step - loss: 126.0550
Epoch 10/25
5000/5000 [==============================] - 2s 454us/step - loss: 125.1464
Epoch 11/25
5000/5000 [==============================] - 2s 453us/step - loss: 124.6182
Epoch 12/25
5000/5000 [==============================] - 2s 450us/step - loss: 124.4888
Epoch 13/25
5000/5000 [==============================] - 2s 454us/step - loss: 124.3022
Epoch 14/25
5000/5000 [==============================] - 2s 453us/step - loss: 124.2704
Epoch 15/25
5000/5000 [==============================] - 2s 454us/step - loss: 124.4872
Epoch 16/25
5000/5000 [==============================] - 2s 453us/step - loss: 123.9926
Epoch 17/25
5000/5000 [==============================] - 2s 452us/step - loss: 124.5247
Epoch 18/25
5000/5000 [==============================] - 2s 457us/step - loss: 124.6059
Epoch 19/25
5000/5000 [==============================] - 2s 453us/step - loss: 123.9300
Epoch 20/25
5000/5000 [==============================] - 2s 451us/step - loss: 124.0661
Epoch 21/25
5000/5000 [==============================] - 2s 454us/step - loss: 124.7735
Epoch 22/25
5000/5000 [==============================] - 2s 451us/step - loss: 124.1924
Epoch 23/25
5000/5000 [==============================] - 2s 451us/step - loss: 124.0452
Epoch 24/25
5000/5000 [==============================] - 2s 457us/step - loss: 124.7132
Epoch 25/25
5000/5000 [==============================] - 2s 456us/step - loss: 124.2815
mse lin simec: 15.78597
mse lin simec pred: 13.13130
Epoch 1/25
5000/5000 [==============================] - 4s 776us/step - loss: 0.0808
Epoch 2/25
5000/5000 [==============================] - 2s 484us/step - loss: 0.0634
Epoch 3/25
5000/5000 [==============================] - 2s 480us/step - loss: 0.0520
Epoch 4/25
5000/5000 [==============================] - 2s 478us/step - loss: 0.0470
Epoch 5/25
5000/5000 [==============================] - 2s 477us/step - loss: 0.0435
Epoch 6/25
5000/5000 [==============================] - 2s 481us/step - loss: 0.0407
Epoch 7/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0383
Epoch 8/25
5000/5000 [==============================] - 2s 477us/step - loss: 0.0360
Epoch 9/25
5000/5000 [==============================] - 2s 478us/step - loss: 0.0343
Epoch 10/25
5000/5000 [==============================] - 2s 478us/step - loss: 0.0326
Epoch 11/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0311
Epoch 12/25
5000/5000 [==============================] - 2s 479us/step - loss: 0.0303
Epoch 13/25
5000/5000 [==============================] - 2s 479us/step - loss: 0.0298
Epoch 14/25
5000/5000 [==============================] - 2s 478us/step - loss: 0.0291
Epoch 15/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0287
Epoch 16/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0283
Epoch 17/25
5000/5000 [==============================] - 2s 481us/step - loss: 0.0279
Epoch 18/25
5000/5000 [==============================] - 2s 476us/step - loss: 0.0276
Epoch 19/25
5000/5000 [==============================] - 2s 483us/step - loss: 0.0273
Epoch 20/25
5000/5000 [==============================] - 2s 479us/step - loss: 0.0267
Epoch 21/25
5000/5000 [==============================] - 2s 480us/step - loss: 0.0267
Epoch 22/25
5000/5000 [==============================] - 2s 478us/step - loss: 0.0266
Epoch 23/25
5000/5000 [==============================] - 2s 478us/step - loss: 0.0263
Epoch 24/25
5000/5000 [==============================] - 2s 481us/step - loss: 0.0258
Epoch 25/25
5000/5000 [==============================] - 2s 473us/step - loss: 0.0257
mse class simec: 0.02732
mse class simec pred: 0.02262
noise dimensions: 6272
mse lin kpca: 336.01312
Epoch 1/25
5000/5000 [==============================] - 4s 754us/step - loss: 938.4899
Epoch 2/25
5000/5000 [==============================] - 2s 456us/step - loss: 713.5829
Epoch 3/25
5000/5000 [==============================] - 2s 456us/step - loss: 451.8191
Epoch 4/25
5000/5000 [==============================] - 2s 455us/step - loss: 302.1120
Epoch 5/25
5000/5000 [==============================] - 2s 454us/step - loss: 208.9502
Epoch 6/25
5000/5000 [==============================] - 2s 457us/step - loss: 158.3941
Epoch 7/25
5000/5000 [==============================] - 2s 457us/step - loss: 136.7988
Epoch 8/25
5000/5000 [==============================] - 2s 460us/step - loss: 129.4654
Epoch 9/25
5000/5000 [==============================] - 2s 455us/step - loss: 126.4024
Epoch 10/25
5000/5000 [==============================] - 2s 451us/step - loss: 126.3072
Epoch 11/25
5000/5000 [==============================] - 2s 456us/step - loss: 125.5732
Epoch 12/25
5000/5000 [==============================] - 2s 455us/step - loss: 125.4927
Epoch 13/25
5000/5000 [==============================] - 2s 455us/step - loss: 125.6391
Epoch 14/25
5000/5000 [==============================] - 2s 454us/step - loss: 125.1141
Epoch 15/25
5000/5000 [==============================] - 2s 457us/step - loss: 125.3527
Epoch 16/25
5000/5000 [==============================] - 2s 456us/step - loss: 126.0021
Epoch 17/25
5000/5000 [==============================] - 2s 457us/step - loss: 125.0300
Epoch 18/25
5000/5000 [==============================] - 2s 457us/step - loss: 126.4445
Epoch 19/25
5000/5000 [==============================] - 2s 455us/step - loss: 124.2469
Epoch 20/25
5000/5000 [==============================] - 2s 456us/step - loss: 124.0898
Epoch 21/25
5000/5000 [==============================] - 2s 456us/step - loss: 125.7089
Epoch 22/25
5000/5000 [==============================] - 2s 456us/step - loss: 125.0004
Epoch 23/25
5000/5000 [==============================] - 2s 458us/step - loss: 126.0220
Epoch 24/25
5000/5000 [==============================] - 2s 458us/step - loss: 124.7127
Epoch 25/25
5000/5000 [==============================] - 2s 458us/step - loss: 125.5696
mse lin simec: 16.50026
mse lin simec pred: 13.81132
Epoch 1/25
5000/5000 [==============================] - 4s 790us/step - loss: 0.0867
Epoch 2/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0722
Epoch 3/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0614
Epoch 4/25
5000/5000 [==============================] - 2s 480us/step - loss: 0.0531
Epoch 5/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0478
Epoch 6/25
5000/5000 [==============================] - 2s 480us/step - loss: 0.0447
Epoch 7/25
5000/5000 [==============================] - 2s 480us/step - loss: 0.0416
Epoch 8/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0394
Epoch 9/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0377
Epoch 10/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0353
Epoch 11/25
5000/5000 [==============================] - 2s 481us/step - loss: 0.0338
Epoch 12/25
5000/5000 [==============================] - 2s 479us/step - loss: 0.0322
Epoch 13/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0316
Epoch 14/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0313
Epoch 15/25
5000/5000 [==============================] - 2s 482us/step - loss: 0.0303
Epoch 16/25
5000/5000 [==============================] - 2s 488us/step - loss: 0.0302
Epoch 17/25
5000/5000 [==============================] - 2s 488us/step - loss: 0.0299
Epoch 18/25
5000/5000 [==============================] - 2s 483us/step - loss: 0.0294
Epoch 19/25
5000/5000 [==============================] - 2s 481us/step - loss: 0.0290
Epoch 20/25
5000/5000 [==============================] - 2s 487us/step - loss: 0.0285
Epoch 21/25
5000/5000 [==============================] - 2s 481us/step - loss: 0.0282
Epoch 22/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0284
Epoch 23/25
5000/5000 [==============================] - 2s 486us/step - loss: 0.0280
Epoch 24/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0278
Epoch 25/25
5000/5000 [==============================] - 2s 487us/step - loss: 0.0273
mse class simec: 0.02964
mse class simec pred: 0.02406
noise dimensions: 7840
mse lin kpca: 509.91721
Epoch 1/25
5000/5000 [==============================] - 4s 762us/step - loss: 942.9726
Epoch 2/25
5000/5000 [==============================] - 2s 463us/step - loss: 728.0957
Epoch 3/25
5000/5000 [==============================] - 2s 456us/step - loss: 473.5573
Epoch 4/25
5000/5000 [==============================] - 2s 461us/step - loss: 308.7493
Epoch 5/25
5000/5000 [==============================] - 2s 473us/step - loss: 217.7473
Epoch 6/25
5000/5000 [==============================] - 2s 468us/step - loss: 175.2631
Epoch 7/25
5000/5000 [==============================] - 2s 468us/step - loss: 156.2391
Epoch 8/25
5000/5000 [==============================] - 2s 467us/step - loss: 146.0503
Epoch 9/25
5000/5000 [==============================] - 2s 464us/step - loss: 140.2577
Epoch 10/25
5000/5000 [==============================] - 2s 479us/step - loss: 135.4399
Epoch 11/25
5000/5000 [==============================] - 2s 482us/step - loss: 130.3495
Epoch 12/25
5000/5000 [==============================] - 2s 472us/step - loss: 128.2118
Epoch 13/25
5000/5000 [==============================] - 2s 466us/step - loss: 126.4001
Epoch 14/25
5000/5000 [==============================] - 2s 468us/step - loss: 126.6032
Epoch 15/25
5000/5000 [==============================] - 2s 464us/step - loss: 127.1709
Epoch 16/25
5000/5000 [==============================] - 2s 477us/step - loss: 126.9454
Epoch 17/25
5000/5000 [==============================] - 2s 471us/step - loss: 126.2727
Epoch 18/25
5000/5000 [==============================] - 2s 469us/step - loss: 127.3869
Epoch 19/25
5000/5000 [==============================] - 2s 467us/step - loss: 126.8581
Epoch 20/25
5000/5000 [==============================] - 2s 462us/step - loss: 127.3288
Epoch 21/25
5000/5000 [==============================] - 2s 468us/step - loss: 125.9762
Epoch 22/25
5000/5000 [==============================] - 2s 462us/step - loss: 127.3586
Epoch 23/25
5000/5000 [==============================] - 2s 462us/step - loss: 126.1602
Epoch 24/25
5000/5000 [==============================] - 2s 463us/step - loss: 126.4526
Epoch 25/25
5000/5000 [==============================] - 2s 461us/step - loss: 127.6211
mse lin simec: 26.53421
mse lin simec pred: 17.41803
Epoch 1/25
5000/5000 [==============================] - 4s 807us/step - loss: 0.0906
Epoch 2/25
5000/5000 [==============================] - 2s 486us/step - loss: 0.0787
Epoch 3/25
5000/5000 [==============================] - 2s 489us/step - loss: 0.0709
Epoch 4/25
5000/5000 [==============================] - 3s 505us/step - loss: 0.0623
Epoch 5/25
5000/5000 [==============================] - 2s 491us/step - loss: 0.0547
Epoch 6/25
5000/5000 [==============================] - 2s 494us/step - loss: 0.0489
Epoch 7/25
5000/5000 [==============================] - 2s 495us/step - loss: 0.0455
Epoch 8/25
5000/5000 [==============================] - 2s 492us/step - loss: 0.0429
Epoch 9/25
5000/5000 [==============================] - 2s 489us/step - loss: 0.0412
Epoch 10/25
5000/5000 [==============================] - 2s 493us/step - loss: 0.0384
Epoch 11/25
5000/5000 [==============================] - 2s 487us/step - loss: 0.0370
Epoch 12/25
5000/5000 [==============================] - 3s 500us/step - loss: 0.0358
Epoch 13/25
5000/5000 [==============================] - 3s 500us/step - loss: 0.0355
Epoch 14/25
5000/5000 [==============================] - 2s 489us/step - loss: 0.0346
Epoch 15/25
5000/5000 [==============================] - 2s 497us/step - loss: 0.0340
Epoch 16/25
5000/5000 [==============================] - 2s 498us/step - loss: 0.0336
Epoch 17/25
5000/5000 [==============================] - 2s 499us/step - loss: 0.0332
Epoch 18/25
5000/5000 [==============================] - 2s 485us/step - loss: 0.0323
Epoch 19/25
5000/5000 [==============================] - 2s 487us/step - loss: 0.0318
Epoch 20/25
5000/5000 [==============================] - 2s 494us/step - loss: 0.0316
Epoch 21/25
5000/5000 [==============================] - 2s 489us/step - loss: 0.0314
Epoch 22/25
5000/5000 [==============================] - 2s 491us/step - loss: 0.0313
Epoch 23/25
5000/5000 [==============================] - 2s 490us/step - loss: 0.0308
Epoch 24/25
5000/5000 [==============================] - 2s 492us/step - loss: 0.0305
Epoch 25/25
5000/5000 [==============================] - 3s 501us/step - loss: 0.0303
mse class simec: 0.03541
mse class simec pred: 0.02689
mse_lin_kpca= [11.048889013360812, 11.373927011271753, 24.909563639794129, 89.558068265218964, 199.01192771624278, 336.01311710236365, 509.91720904720779]
mse_lin_simec= [11.067329893150051, 11.171933783584489, 11.481151194808788, 12.355816088572375, 15.785973979140197, 16.500255982706136, 26.534214593301577]
mse_class_simec= [0.022975397782250502, 0.022738621182952039, 0.023163301485489247, 0.024827144074323997, 0.027324134083521311, 0.029640995068240038, 0.035409908556008077]
mse_lin_simec_pred= [11.058381870431411, 11.110588094397116, 11.240437003701189, 11.717718070203658, 13.131295376522827, 13.811317734238354, 17.418031038628076]
mse_class_simec_pred= [0.020425897729821618, 0.020199900200684656, 0.020487262080180067, 0.021453983766734068, 0.022622070392637701, 0.024057664812919637, 0.026892399817994492]






    Out[6]:





Text(0,0.5,u'Mean Squared Error of $\\hat{S}$')



In [ ]: