Consistency and Robustness Analysis

This notebook analyzes and plots data generated by two python scripts to perform "consistency" and "robustness" analysis. Consistency refers to the ability of the model to reproduce similar results under different random instantiations of the network according to the same probabilistic constraints (i.e. synapse formation probability). Robustness analysis refers to the ability of the model to reproduce similar results when the parameters governing the model are slightly perturbed. Two sections below investigate this.

Consistency Analysis

The purpose of this section is to analyze and plot the results of multiple simulations of the MLI-PKJ network with the same parameters but different random seeds. We want to see how consistent the resulting activities of the network are among different simulations. A separate python file, '../experiments/MLI_PKJ_net_consistency_analysis.py' generates the data that is analyzed here.



In [ ]:

    
import cPickle
sys.path.append('../../')
from MLI_PKJ_net.plotting_util import *



In [ ]:

    
# load the results from disk, there are 100 simulation results by default
in_dir = '../../results/'
results = cPickle.load(open(in_dir+'consistency_results.pkl'))

# shuffle the order of the simulation results
shuffle(results)

Plot MLI firing rates and ISI CV from four simulations



In [ ]:

    
fig = figure(figsize=(12,8),dpi=300)
colors = ['r','b','g','k']

# choose four random simulations to plot
for i,r in enumerate(results[:4]):
    
    # firing rate and ISI CV statistics from a simulation by population
    mli_fr, mli_cv, pkj_fr, pkj_cv = r
    
    # for MLIs only
    frs = zip(*mli_fr)[1]
    cvs = zip(*mli_cv)[1]
    
    # add these results to a scatter plot, each simulation gets a different color
    scatter(frs,cvs,s=80, c=colors[i], marker='+')

# label the plot
xlabel('Mean Neuron Firing Rate (Hz)',fontsize=18)
ylabel('Mean ISI CV', fontsize=18)
xticks(fontsize=18)
yticks(fontsize=18)
xlim([0,30])
ylim([.1,1.4])
title('MLI Firing Rate vs ISI CV Across Simulations',fontsize=20)

Plot PKJ firing rates and ISI CV from four simulations



In [ ]:

    
fig = figure(figsize=(12,8))
colors = ['r','b','g','k']
for i,r in enumerate(results[:4]):
    mli_fr, mli_cv, pkj_fr, pkj_cv = r
    frs = zip(*pkj_fr)[1]
    cvs = zip(*pkj_cv)[1]
    scatter(frs,cvs,s=80, c=colors[i], marker='+')

xlabel('Mean Neuron Firing Rate (Hz)',fontsize=18)
ylabel('Mean ISI CV', fontsize=18)
xticks(fontsize=16)
yticks(fontsize=16)
xlim([10,33])
ylim([.2,.56])
title('PKJ Firing Rate vs ISI CV Across Simulations',fontsize=18)

Robustness Analysis

The purpose of this section to analyze and plot the results of multiple simulations of the MLI-PKJ network with slight perturbations to the parameters. We want to see how robust the resulting activities of the network are among different simulations. A separate python file, '../experiments/MLI_PKJ_net_robustness_analysis.py' generates the data that is analyzed here.

In this analysis, we will compare the distribution of MLI Firing rates, MLI ISI CVs, PKJ Firing rates, PKJ ISI CVs in two conditions: when the parameters are unperturbed and when the parameters are randomly perturbed each simulation.



In [ ]:

    
def population_means(results):
    '''
    compute the average values across the entire population of neuron by neuron type for
    firing rates and ISI CVs. The input data is the average firing rate and ISI CV for each
    neuron during a simulation.
    '''
    mean_mli_frs, mean_mli_cvs = [], []
    mean_pkj_frs, mean_pkj_cvs = [], []
    for r in results:
        mli_fr, mli_cv, pkj_fr, pkj_cv = r
        mean_mli_frs.append(mean(zip(*mli_fr)[1]))
        mean_mli_cvs.append(mean([cv for cv in zip(*mli_cv)[1] if not isnan(cv)])) # some neurons failed to spike creating ISI CV values of NaN
        mean_pkj_frs.append(mean(zip(*pkj_fr)[1]))
        mean_pkj_cvs.append(mean([cv for cv in zip(*pkj_cv)[1] if not isnan(cv)]))
    return mean_mli_frs, mean_mli_cvs, mean_pkj_frs, mean_pkj_cvs



In [ ]:

    
# load data from network with unperturbed parameters over 100 simulations
consistency_results = cPickle.load(open(in_dir+'consistency_results.pkl'))
c_mean_mli_frs, c_mean_mli_cvs, c_mean_pkj_frs, c_mean_pkj_cvs = population_means(consistency_results)

# load data from network with randomly perturbed parameters over 100 simulations
robustness_results = cPickle.load(open(in_dir+'robustness_results.pkl'))
r_mean_mli_frs, r_mean_mli_cvs, r_mean_pkj_frs, r_mean_pkj_cvs = population_means(robustness_results)

# create figure
fig = figure(figsize(14,10), dpi=300)

# Histograms of MLI population mean firing rates for each scenario
ax = fig.add_subplot(221)
ax.hist(array(c_mean_mli_frs),alpha=.4,color='r')
ax.hist(array(r_mean_mli_frs),alpha=.4, color='#3299BB')
simpleaxis(ax)
tick_params(labelsize=16)
xlabel('MLI Population Mean Firing Rate (Hz)',fontsize=16)
ylabel('Count',fontsize=18)

# Histograms of MLI population mean ISI CVs for each scenario
ax = fig.add_subplot(222)
ax.hist(array(c_mean_mli_cvs),alpha=.4,color='r')
ax.hist(array(r_mean_mli_cvs),alpha=.4, color='#3299BB')
simpleaxis(ax)
tick_params(labelsize=16)
xlabel('MLI Population Mean ISI CV',fontsize=16)
ylabel('Count',fontsize=18)

# Histograms of PKJ population mean firing rates for each scenario
ax = fig.add_subplot(223)
ax.hist(array(c_mean_pkj_frs),alpha=.4,color='r')
ax.hist(array(r_mean_pkj_frs),alpha=.4, color='#3299BB')
simpleaxis(ax)
tick_params(labelsize=16)
xlabel('PKJ Population Mean Firing Rate (Hz)',fontsize=16)
ylabel('Count',fontsize=18)

# Histograms of PKJ population mean ISI CVs for each scenario
ax = fig.add_subplot(224)
ax.hist(array(c_mean_pkj_cvs),alpha=.4,color='r')
ax.hist(array(r_mean_pkj_cvs),alpha=.4, color='#3299BB')
simpleaxis(ax)
tick_params(labelsize=16)
xlabel('PKJ Population Mean ISI CV',fontsize=16)
ylabel('Count',fontsize=18)
tight_layout()



In [ ]:



In [ ]: