Results 9:45 PM

The following notebook does the following:

  • 3 maps to be made:
    • sign of c1-c2 * (1-q) value
    • p value map
    • average c1-c2

Check consistency of signs


  • Excel:

    • For each ROI I have to report minimum abs p value (among all voxels) and min q value and number of voxels for which abs q value les than 0.1, 0.05, 0.01

    • And number of voxels for absolute p value les than 0.1, 0.05, 0.01


In [3]:
import numpy as np
from scipy import stats
import os
from os.path import join as opj
from multiprocessing import Pool
import multiprocessing.managers
from functools import partial
import nibabel as nib
import params
# List of functional files
# fc_file_list = np.asarray(['/home1/varunk/result/functionalConnectivity/_subject_id_0050952/pearcoff/0050952_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050953/pearcoff/0050953_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050954/pearcoff/0050954_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050955/pearcoff/0050955_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050956/pearcoff/0050956_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050957/pearcoff/0050957_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050958/pearcoff/0050958_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050959/pearcoff/0050959_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050960/pearcoff/0050960_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050961/pearcoff/0050961_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050962/pearcoff/0050962_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050964/pearcoff/0050964_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050965/pearcoff/0050965_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050966/pearcoff/0050966_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050967/pearcoff/0050967_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050968/pearcoff/0050968_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050969/pearcoff/0050969_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050970/pearcoff/0050970_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050971/pearcoff/0050971_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050972/pearcoff/0050972_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050973/pearcoff/0050973_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050974/pearcoff/0050974_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050975/pearcoff/0050975_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050976/pearcoff/0050976_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050977/pearcoff/0050977_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050978/pearcoff/0050978_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050979/pearcoff/0050979_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050980/pearcoff/0050980_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050981/pearcoff/0050981_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050982/pearcoff/0050982_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050983/pearcoff/0050983_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050984/pearcoff/0050984_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050985/pearcoff/0050985_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050986/pearcoff/0050986_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050987/pearcoff/0050987_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050988/pearcoff/0050988_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050989/pearcoff/0050989_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050990/pearcoff/0050990_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050991/pearcoff/0050991_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050992/pearcoff/0050992_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050993/pearcoff/0050993_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050994/pearcoff/0050994_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050995/pearcoff/0050995_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050996/pearcoff/0050996_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050997/pearcoff/0050997_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050998/pearcoff/0050998_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0050999/pearcoff/0050999_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051000/pearcoff/0051000_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051001/pearcoff/0051001_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051002/pearcoff/0051002_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051003/pearcoff/0051003_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051006/pearcoff/0051006_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051007/pearcoff/0051007_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051008/pearcoff/0051008_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051009/pearcoff/0051009_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051010/pearcoff/0051010_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051011/pearcoff/0051011_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051012/pearcoff/0051012_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051013/pearcoff/0051013_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051014/pearcoff/0051014_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051015/pearcoff/0051015_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051016/pearcoff/0051016_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051017/pearcoff/0051017_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051018/pearcoff/0051018_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051019/pearcoff/0051019_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051020/pearcoff/0051020_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051021/pearcoff/0051021_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051023/pearcoff/0051023_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051024/pearcoff/0051024_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051025/pearcoff/0051025_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051026/pearcoff/0051026_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051027/pearcoff/0051027_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051028/pearcoff/0051028_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051029/pearcoff/0051029_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051030/pearcoff/0051030_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051032/pearcoff/0051032_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051033/pearcoff/0051033_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051034/pearcoff/0051034_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051035/pearcoff/0051035_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051036/pearcoff/0051036_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051038/pearcoff/0051038_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051039/pearcoff/0051039_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051040/pearcoff/0051040_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051041/pearcoff/0051041_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051042/pearcoff/0051042_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051044/pearcoff/0051044_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051045/pearcoff/0051045_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051046/pearcoff/0051046_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051047/pearcoff/0051047_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051048/pearcoff/0051048_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051049/pearcoff/0051049_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051050/pearcoff/0051050_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051051/pearcoff/0051051_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051052/pearcoff/0051052_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051053/pearcoff/0051053_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051054/pearcoff/0051054_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051055/pearcoff/0051055_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051056/pearcoff/0051056_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051057/pearcoff/0051057_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051058/pearcoff/0051058_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051059/pearcoff/0051059_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051060/pearcoff/0051060_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051061/pearcoff/0051061_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051062/pearcoff/0051062_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051063/pearcoff/0051063_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051064/pearcoff/0051064_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051065/pearcoff/0051065_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051066/pearcoff/0051066_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051067/pearcoff/0051067_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051068/pearcoff/0051068_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051069/pearcoff/0051069_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051070/pearcoff/0051070_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051071/pearcoff/0051071_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051072/pearcoff/0051072_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051073/pearcoff/0051073_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051074/pearcoff/0051074_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051075/pearcoff/0051075_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051076/pearcoff/0051076_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051077/pearcoff/0051077_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051078/pearcoff/0051078_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051079/pearcoff/0051079_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051080/pearcoff/0051080_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051081/pearcoff/0051081_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051082/pearcoff/0051082_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051083/pearcoff/0051083_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051084/pearcoff/0051084_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051085/pearcoff/0051085_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051086/pearcoff/0051086_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051087/pearcoff/0051087_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051088/pearcoff/0051088_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051089/pearcoff/0051089_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051090/pearcoff/0051090_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051091/pearcoff/0051091_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051093/pearcoff/0051093_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051094/pearcoff/0051094_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051095/pearcoff/0051095_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051096/pearcoff/0051096_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051097/pearcoff/0051097_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051098/pearcoff/0051098_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051099/pearcoff/0051099_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051100/pearcoff/0051100_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051101/pearcoff/0051101_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051102/pearcoff/0051102_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051103/pearcoff/0051103_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051104/pearcoff/0051104_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051105/pearcoff/0051105_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051106/pearcoff/0051106_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051107/pearcoff/0051107_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051108/pearcoff/0051108_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051109/pearcoff/0051109_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051110/pearcoff/0051110_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051111/pearcoff/0051111_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051112/pearcoff/0051112_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051113/pearcoff/0051113_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051114/pearcoff/0051114_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051115/pearcoff/0051115_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051116/pearcoff/0051116_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051117/pearcoff/0051117_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051118/pearcoff/0051118_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051119/pearcoff/0051119_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051120/pearcoff/0051120_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051121/pearcoff/0051121_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051122/pearcoff/0051122_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051123/pearcoff/0051123_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051124/pearcoff/0051124_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051125/pearcoff/0051125_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051126/pearcoff/0051126_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051127/pearcoff/0051127_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051128/pearcoff/0051128_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051129/pearcoff/0051129_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051130/pearcoff/0051130_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051131/pearcoff/0051131_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051146/pearcoff/0051146_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051147/pearcoff/0051147_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051148/pearcoff/0051148_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051149/pearcoff/0051149_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051150/pearcoff/0051150_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051151/pearcoff/0051151_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051152/pearcoff/0051152_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051153/pearcoff/0051153_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051154/pearcoff/0051154_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051155/pearcoff/0051155_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051156/pearcoff/0051156_fc_map.npy', '/home1/varunk/result/functionalConnectivity/_subject_id_0051159/pearcoff/0051159_fc_map.npy'])

if not params.run_all:
    fc_file_list = list(np.load('/home1/varunk/result/functionalConnectivity/save_fc_file_list/fc_file_list.npy'))

In [4]:
# Paths
if not params.run_all:
    path_cwd = os.getcwd()
    path_split_list = path_cwd.split('/')
    s = path_split_list[0:-1]
    s = opj('/',*s) # *s converts list to path, # also it is very important to add '/' in the begining so it is read as directory later
    base_directory = opj(s,'result')
    # data_directory = opj(s,data_path)
    wf_name = "functionalConnectivity"

In [5]:
# For passing on the shared file:
class MyManager(multiprocessing.managers.BaseManager):
    pass
MyManager.register('np_zeros', np.zeros, multiprocessing.managers.ArrayProxy)

In [ ]:
fc_file_list # Array that contains the list of paths to fc files

In [ ]:
number_of_fcmaps = len(fc_file_list) #184

In [ ]:
number_of_fcmaps

In [ ]:
%%time
# This method reads the significant matrix again and again and stors the moving sum and then at last find the average
from scipy import stats
import os
from os.path import join as opj
# from multiprocessing import Pool
# import multiprocessing.managers
# from functools import partial

number_of_ROIs, number_of_voxels = np.load(fc_file_list[0], mmap_mode='r').shape

autism_mean = np.zeros((number_of_ROIs, number_of_voxels)) # average c1
td_mean = np.zeros((number_of_ROIs, number_of_voxels)) # average c2

if not params.run_all:
    autism_start_index = 0
    autism_end_index = 78
    td_start_index = 79
    td_end_index = 157 #183
    autism_subject_indexes = list(np.arange(autism_start_index, autism_end_index + 1))
    td_subject_indexes = list(np.arange(td_start_index, td_end_index + 1))


# For calculating mean autism i.e. C1
for participant_index in autism_subject_indexes:
    print('Autistic Participant index:',participant_index)
    autism_mean = autism_mean + np.load(fc_file_list[participant_index]) 

# number_of_autistic =  autism_end_index - autism_start_index + 1
number_of_autistic =  len(autism_subject_indexes)

autism_mean = np.divide(autism_mean, number_of_autistic)

# For calculating mean TD i.e. C2
for participant_index in td_subject_indexes:
    print('TD Participant index ',participant_index)
    td_mean = td_mean + np.load(fc_file_list[participant_index]) 

# number_of_td =  td_end_index - td_start_index + 1 
number_of_td = len(td_subject_indexes)

td_mean = np.divide(td_mean, number_of_td)

In [6]:
save_destination = opj(base_directory,wf_name,'results')
if not os.path.exists(save_destination):
    os.mkdir(save_destination)


---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-6-f70c639829ee> in <module>()
----> 1 save_destination = opj(base_directory,wf_name,'results')
      2 if not os.path.exists(save_destination):
      3     os.mkdir(save_destination)
      4 

NameError: name 'base_directory' is not defined

In [ ]:
autism_mean_path = opj(save_destination,'autism_mean')
td_mean_path = opj(save_destination,'td_mean')



np.save(autism_mean_path,autism_mean)
np.save(td_mean_path,td_mean)

In [ ]:


In [ ]:
autismMinusTd = autism_mean - td_mean

In [ ]:
td_mean_path = opj(save_destination,'autismMinusTd')

np.save(td_mean_path,autismMinusTd)

In [ ]:
np.min(autismMinusTd)

In [ ]:
np.min(autism_mean), np.max(autism_mean)

File Descriptions

  • Q file name -> fdr_brain_voxel_list.npy
  • P values file -> significance_matrix.npy
  • C1 File -> autism_mean.npy
  • C2 File -> td_meaan.npy
  • C1-C2 File -> autismMinusTd.npy

In [4]:
# Check if the values are saved nicely

# P = np.load('/home1/varunk/Autism-Connectome-Analysis-bids-related/significance/significance_matrix.npy',mmap_mode='r')
# Q = np.load('/home1/varunk/Autism-Connectome-Analysis-bids-related/fdr_brain_voxel_list.npy',mmap_mode='r')
# C1MinusC2 = np.load('/home1/varunk/Autism-Connectome-Analysis-bids-related/autismMinusTd.npy',mmap_mode='r')

C1MinusC2_file = '/home1/varunk/Autism-Connectome-Analysis-bids-related/autismMinusTd.npy'


if not params.run_all:
    Pvals_file_path = '/home1/varunk/Autism-Connectome-Analysis-bids-related/significance/significance_matrix.npy'
    Qvals_file_path = '/home1/varunk/Autism-Connectome-Analysis-bids-related/fdr_brain_voxel_list.npy'

P = np.load(Pvals_file_path,mmap_mode='r')
Q = np.load(Qvals_file_path,mmap_mode='r')
C1MinusC2 = np.load(C1MinusC2_file,mmap_mode='r')



# Y = np.load('/home1/varunk/Autism-Connectome-Analysis-bids-related/significance_matrix_wo_z_standardization.npy', mmap_mode='r')

In [6]:
# c1-c2 * -1 * log10(q)
map1 = np.multiply(np.sign(C1MinusC2),(-1*np.log10(Q)))

In [5]:
np.where(C1MinusC2 == 0)


Out[5]:
(array([], dtype=int64), array([], dtype=int64))

In [24]:
np.min(map1), np.max(map1)


Out[24]:
(-0.22085895627840782, 0.29561788719604293)

In [ ]:


In [12]:
map2 = np.multiply(np.sign(C1MinusC2),(-1*np.log10(P)))

In [6]:
map3 = C1MinusC2

In [7]:
num_ROIs = P.shape[0]

In [ ]:
# roi_idx = 200
# np.min(X[roi_idx,:]),np.max(X[roi_idx,:])

The following function creates a functional brain file out of the given ROI x Voxel matrix

Inputs: ROI x Voxel matrix
Output: Functional file with ROI as t-dimension


In [ ]:
# Read mask files:
mask_file = os.path.expandvars('$FSLDIR/data/standard/MNI152_T1_2mm_brain_mask.nii.gz')
mask_data = nib.load(mask_file)
mask = mask_data.get_data()

# read the file on which you want to write the p-values
brain_file = '/home1/varunk/data/ABIDE-BIDS-Preprocessed/NYU/sub-0050952/func/sub-0050952_task-rest_run-1_bold.nii.gz'
brain_data = nib.load(brain_file)
brain = brain_data.get_data()

# Edit header to store ROI correlation maps in t(4th) dimension

(brain_data.header).set_data_shape([91,109,91,num_ROIs])

In [ ]:
def convert_to_brain(brain_roi_tensor, brain_voxel_list):
    mask_data = nib.load(mask_file)
    mask = mask_data.get_data()
    
    # to use the deader of an anatomical file tht has float datatype. mask had uint datatype that made all the p-values zero
    brain_file = '/home1/varunk/data/ABIDE-BIDS-Preprocessed/NYU/sub-0050952/anat/sub-0050952_T1w.nii.gz'
    brain_data = nib.load(brain_file)
    
    
    # Empty brain to store the voxel values
    brain = brain_data.get_data()
    
    brain_corrected_mask = np.zeros((brain.shape))


    roi_number = int(brain_voxel_list[0]) - 1
    brain_voxel_list = brain_voxel_list[1:]
    
    x_dim, y_dim, z_dim = mask.shape

    brain_voxel_counter = 0
    for i in range(x_dim):
        for j in range(y_dim):
            for k in range(z_dim):
                if mask[i,j,k] == 1:
                    brain[i,j,k] = brain_voxel_list[brain_voxel_counter]
                    brain_voxel_counter = brain_voxel_counter + 1
                    

    
    
    if brain_voxel_counter == len(brain_voxel_list):
        
     
    
        brain_roi_tensor[:,:,:,roi_number] = brain
        
        print('Stored ROI file the shared arrays! ')
        print("Job Done for ROI-",roi_number)
      
        return 1
    else:
        print("Voxels fitting failed for ROI-",roi_number)
        return 0

In [ ]:
%%time
# Input roi-voxel matrix
for i,X in enumerate([map1,map2,map3]):
    X = np.c_[1:X.shape[0]+1,X] # Add ROI number

    # Create pool of 8 workers

    pool_inputs = [] #np.arange(number_of_ROIs)

    for roi in range(num_ROIs):
        pool_inputs.append(X[roi,:])


    # print (brain_data.header)


    m = MyManager()
    m.start()
    brain_roi_tensor = m.np_zeros((brain_data.header.get_data_shape()))
    # fdrcorrected_brain_4Dtensor_mask = m.np_zeros((brain_data.header.get_data_shape()))

    func = partial(convert_to_brain, brain_roi_tensor)

    pool = Pool(8)

    data_outputs = pool.map(func, pool_inputs)

    brain_roi_tensor = np.asarray(brain_roi_tensor)

    # Saving the brains: 
    brain_with_header = nib.Nifti1Image(brain_roi_tensor, affine=brain_data.affine,header = brain_data.header)
    
    map_name = 'map'+ str(i+1) + '.nii.gz'
    map_path = opj(save_destination,map_name)

    nib.save(brain_with_header,map_path)

In [ ]:
# for i,j in enumerate(['q','w','r']):
#     print(i,j)

Excel Work

Now, do the following:

For each ROI I have to report minimum abs p value (among all voxels) and min q value and number of voxels for which abs q value les than 0.1, 0.05, 0.01

And number of voxels for absolute p value les than 0.1, 0.05, 0.01


In [10]:
def count_voxel_stats(roi_voxel_stats_matrix,Pval_Qval_tuple):
    

    P_brain_voxel_list, Q_brain_voxel_list = Pval_Qval_tuple

    roi_number = int(P_brain_voxel_list[0]) - 1
    P_brain_voxel_list = P_brain_voxel_list[1:]
    
    
#     min p value 
    min_pval = np.min(P_brain_voxel_list)
    
#     min q value
    min_qval = np.min(Q_brain_voxel_list)
    
#     p value less than 0.1
    p_lt_point_1 = np.shape(np.where(P_brain_voxel_list < 0.1))[1]  
    
#     p value less than 0.01    
    p_lt_point_01 = np.shape(np.where(P_brain_voxel_list < 0.01))[1]
    
#     p value less than 0.05    
    p_lt_point_05 = np.shape(np.where(P_brain_voxel_list < 0.05))[1]

#     p value less than 0.1    
    q_lt_point_1 = np.shape(np.where(Q_brain_voxel_list < 0.1))[1]  

#     p value less than 0.01    
    q_lt_point_01 = np.shape(np.where(Q_brain_voxel_list < 0.01))[1]

#     p value less than 0.05    
    q_lt_point_05 = np.shape(np.where(Q_brain_voxel_list < 0.05))[1]
    
# Store in the roi_voxel_stats_matrix 

    roi_voxel_stats_matrix[roi_number,:] = [min_pval,min_qval,p_lt_point_1,p_lt_point_01,p_lt_point_05,q_lt_point_1,q_lt_point_01,q_lt_point_05]
    
    print('Stats computed for ROI: ',roi_number)
    return 1

In [11]:
%%time

# Input roi-voxel matrix
X = P
Y = Q
X = np.c_[1:X.shape[0]+1,X] # Add ROI number

# Create pool of 8 workers


pool_inputs = [] #np.arange(number_of_ROIs)

for roi in range(num_ROIs):
    inputs = (X[roi,:],np.asarray(Y[roi,:]))
    pool_inputs.append(inputs)
    
# from IPython.core.debugger import Tracer;
# Tracer()()

# print (brain_data.header)

num_of_stat_entries = 8

m = MyManager()
m.start()

roi_voxel_stats_matrix = m.np_zeros((Y.shape[0], num_of_stat_entries ))
# fdrcorrected_brain_4Dtensor_mask = m.np_zeros((brain_data.header.get_data_shape()))

func = partial(count_voxel_stats, roi_voxel_stats_matrix)

pool = Pool(8)

data_outputs = pool.map(func, pool_inputs)


/root/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:17: DeprecationWarning: `Tracer` is deprecated since version 5.1, directly use `IPython.core.debugger.Pdb.set_trace()`
None
> <timed exec>(21)<module>()

ipdb> c
Hello
Reached
Reached
Stats computed for ROI:  0
Stats computed for ROI:  1
Reached
Stats computed for ROI:  2
Reached
Stats computed for ROI:  3
Reached
Stats computed for ROI:  4
Reached
Stats computed for ROI:  5
Reached
Stats computed for ROI:  6
Reached
Reached
Stats computed for ROI:  7
Reached
Stats computed for ROI:  8
Stats computed for ROI:  9
Reached
Stats computed for ROI:  10
Reached
Stats computed for ROI:  11
Reached
Stats computed for ROI:  12
Reached
Stats computed for ROI:  13
Reached
Reached
Stats computed for ROI:  14
Reached
Stats computed for ROI:  15
Reached
Stats computed for ROI:  16
Reached
Stats computed for ROI:  17
Stats computed for ROI:  18
Reached
Stats computed for ROI:  19
Reached
Stats computed for ROI:  20
Reached
Stats computed for ROI:  21
Reached
Stats computed for ROI:  22
Reached
Stats computed for ROI:  23
Reached
Stats computed for ROI:  24
Reached
Stats computed for ROI:  25
Reached
Stats computed for ROI:  26
Reached
Stats computed for ROI:  27
Reached
Stats computed for ROI:  28
Reached
Stats computed for ROI:  29
Reached
Stats computed for ROI:  30
Reached
Stats computed for ROI:  31
Reached
Stats computed for ROI:  32
Reached
Reached
Stats computed for ROI:  33
Reached
Stats computed for ROI:  34
Reached
Stats computed for ROI:  35
Stats computed for ROI:  36
Reached
Reached
Stats computed for ROI:  37
Reached
Stats computed for ROI:  38
Reached
Stats computed for ROI:  39
Reached
Stats computed for ROI:  40
Stats computed for ROI:  41
Reached
Stats computed for ROI:  42
Reached
Stats computed for ROI:  43
Reached
Stats computed for ROI:  44
Reached
Stats computed for ROI:  45
Reached
Stats computed for ROI:  46
Stats computed for ROI:  53
Reached
Stats computed for ROI:  47
Reached
Stats computed for ROI:  48
Reached
Stats computed for ROI:  49
Reached
Stats computed for ROI:  50
Reached
Stats computed for ROI:  51
Reached
Stats computed for ROI:  52
Reached
Reached
Stats computed for ROI:  54
Reached
Stats computed for ROI:  55
Stats computed for ROI:  62
Reached
Stats computed for ROI:  56
Reached
Stats computed for ROI:  57
Reached
Stats computed for ROI:  58
Reached
Stats computed for ROI:  59
Reached
Stats computed for ROI:  60
Reached
Stats computed for ROI:  61
Reached
Reached
Stats computed for ROI:  63
Reached
Stats computed for ROI:  64
Reached
Reached
Stats computed for ROI:  65
Reached
Stats computed for ROI:  66
Reached
Reached
Stats computed for ROI:  67
Reached
Stats computed for ROI:  72
Reached
Stats computed for ROI:  68
Stats computed for ROI:  73
Reached
Stats computed for ROI:  69
Stats computed for ROI:  74
Reached
Reached
Stats computed for ROI:  70
Stats computed for ROI:  75
Reached
Reached
Stats computed for ROI:  71
Stats computed for ROI:  76
Reached
Stats computed for ROI:  77
Reached
Stats computed for ROI:  78
Reached
Stats computed for ROI:  79
Reached
Stats computed for ROI:  80
Reached
Reached
Stats computed for ROI:  81
Reached
Stats computed for ROI:  82
Reached
Stats computed for ROI:  83
Stats computed for ROI:  84
Reached
Stats computed for ROI:  85
Reached
Stats computed for ROI:  86
Reached
Stats computed for ROI:  87
Reached
Stats computed for ROI:  88
Reached
Reached
Stats computed for ROI:  89
Reached
Stats computed for ROI:  90
Reached
Stats computed for ROI:  91
Reached
Stats computed for ROI:  92
Reached
Stats computed for ROI:  93
Stats computed for ROI:  94
Reached
Stats computed for ROI:  95
Reached
Stats computed for ROI:  96
Reached
Stats computed for ROI:  97
Reached
Stats computed for ROI:  98
Reached
Reached
Stats computed for ROI:  99
Reached
Stats computed for ROI:  100
Reached
Stats computed for ROI:  101
Stats computed for ROI:  102
Reached
Stats computed for ROI:  103
Reached
Stats computed for ROI:  104
Reached
Stats computed for ROI:  105
Reached
Stats computed for ROI:  106
Reached
Stats computed for ROI:  107
Reached
Reached
Stats computed for ROI:  108
Reached
Stats computed for ROI:  109
Reached
Stats computed for ROI:  110
Stats computed for ROI:  111
Reached
Stats computed for ROI:  112
Reached
Stats computed for ROI:  113
Reached
Stats computed for ROI:  114
Reached
Stats computed for ROI:  115
Reached
Stats computed for ROI:  116
Reached
Reached
Stats computed for ROI:  117
Reached
Stats computed for ROI:  118
Reached
Stats computed for ROI:  119
Stats computed for ROI:  120
Reached
Stats computed for ROI:  121
Reached
Stats computed for ROI:  122
Reached
Stats computed for ROI:  123
Reached
Stats computed for ROI:  124
Reached
Stats computed for ROI:  125
Reached
Reached
Stats computed for ROI:  126
Reached
Stats computed for ROI:  127
Reached
Stats computed for ROI:  128
Stats computed for ROI:  129
Reached
Stats computed for ROI:  130
Reached
Stats computed for ROI:  131
Reached
Stats computed for ROI:  132
Reached
Stats computed for ROI:  133
Reached
Stats computed for ROI:  134
Reached
Reached
Stats computed for ROI:  135
Reached
Stats computed for ROI:  136
Reached
Stats computed for ROI:  137
Reached
Stats computed for ROI:  138
Stats computed for ROI:  139
Reached
Stats computed for ROI:  140
Reached
Stats computed for ROI:  141
Reached
Stats computed for ROI:  142
Reached
Stats computed for ROI:  143
Reached
Reached
Stats computed for ROI:  144
Reached
Stats computed for ROI:  145
Stats computed for ROI:  146
Reached
Stats computed for ROI:  147
Reached
Stats computed for ROI:  148
Reached
Stats computed for ROI:  149
Reached
Stats computed for ROI:  150
Reached
Stats computed for ROI:  151
Reached
Stats computed for ROI:  152
Reached
Reached
Stats computed for ROI:  153
Reached
Stats computed for ROI:  154
Reached
Stats computed for ROI:  155
Stats computed for ROI:  156
Reached
Stats computed for ROI:  157
Reached
Stats computed for ROI:  158
Reached
Stats computed for ROI:  159
Reached
Stats computed for ROI:  160
Reached
Reached
Stats computed for ROI:  161
Reached
Stats computed for ROI:  162
Reached
Stats computed for ROI:  163
Stats computed for ROI:  164
Reached
Stats computed for ROI:  165
Reached
Stats computed for ROI:  166
Reached
Stats computed for ROI:  167
Reached
Stats computed for ROI:  168
Reached
Stats computed for ROI:  169
Reached
Stats computed for ROI:  170
Reached
Reached
Stats computed for ROI:  171
Reached
Stats computed for ROI:  172
Reached
Stats computed for ROI:  173
Reached
Stats computed for ROI:  174
Stats computed for ROI:  175
Reached
Stats computed for ROI:  176
Reached
Stats computed for ROI:  177
Reached
Stats computed for ROI:  178
Reached
Stats computed for ROI:  179
Reached
Stats computed for ROI:  183
Stats computed for ROI:  180
Reached
Stats computed for ROI:  181
Reached
Stats computed for ROI:  182
Reached
Reached
Stats computed for ROI:  184
Reached
Stats computed for ROI:  185
Reached
Stats computed for ROI:  186
Reached
Stats computed for ROI:  187
Reached
Stats computed for ROI:  188
Reached
Reached
Stats computed for ROI:  189
Reached
Stats computed for ROI:  190
Reached
Stats computed for ROI:  191
Stats computed for ROI:  192
Reached
Stats computed for ROI:  193
Reached
Stats computed for ROI:  194
Reached
Stats computed for ROI:  195
Reached
Stats computed for ROI:  196
Reached
Stats computed for ROI:  197
Reached
Reached
Stats computed for ROI:  198
Reached
Stats computed for ROI:  199
Reached
Stats computed for ROI:  200
Stats computed for ROI:  201
Reached
Stats computed for ROI:  202
Reached
Stats computed for ROI:  203
Reached
Stats computed for ROI:  204
Reached
Stats computed for ROI:  205
Reached
Stats computed for ROI:  206
Reached
Reached
Stats computed for ROI:  207
Reached
Stats computed for ROI:  208
Reached
Stats computed for ROI:  209
Stats computed for ROI:  210
Reached
Stats computed for ROI:  211
Reached
Stats computed for ROI:  212
Reached
Stats computed for ROI:  213
Reached
Stats computed for ROI:  214
Reached
Stats computed for ROI:  215
Reached
Reached
Stats computed for ROI:  216
Reached
Stats computed for ROI:  217
Reached
Stats computed for ROI:  218
Stats computed for ROI:  219
Reached
Stats computed for ROI:  220
Reached
Stats computed for ROI:  221
Reached
Stats computed for ROI:  222
Reached
Stats computed for ROI:  223
Stats computed for ROI:  228
Reached
Stats computed for ROI:  224
Reached
Stats computed for ROI:  225
Reached
Stats computed for ROI:  226
Reached
Stats computed for ROI:  227
Reached
Reached
Stats computed for ROI:  229
Reached
Stats computed for ROI:  230
Reached
Stats computed for ROI:  231
Reached
Stats computed for ROI:  232
Reached
Stats computed for ROI:  233
Reached
Reached
Stats computed for ROI:  234
Reached
Stats computed for ROI:  235
Reached
Stats computed for ROI:  236
Reached
Stats computed for ROI:  237
Stats computed for ROI:  238
Reached
Stats computed for ROI:  239
Reached
Stats computed for ROI:  240
Reached
Stats computed for ROI:  241
Reached
Stats computed for ROI:  242
Reached
Reached
Stats computed for ROI:  243
Reached
Stats computed for ROI:  244
Reached
Stats computed for ROI:  245
Stats computed for ROI:  246
Reached
Stats computed for ROI:  247
Reached
Stats computed for ROI:  248
Reached
Stats computed for ROI:  249
Reached
Stats computed for ROI:  250
Stats computed for ROI:  255
Reached
Stats computed for ROI:  251
Reached
Stats computed for ROI:  252
Reached
Stats computed for ROI:  253
Reached
Stats computed for ROI:  254
Reached
Reached
Stats computed for ROI:  256
Reached
Stats computed for ROI:  257
Reached
Stats computed for ROI:  258
Reached
Stats computed for ROI:  259
Reached
Stats computed for ROI:  260
Reached
Stats computed for ROI:  261
Reached
Stats computed for ROI:  262
Reached
Reached
Stats computed for ROI:  263
Stats computed for ROI:  270
Reached
Reached
Stats computed for ROI:  264
Stats computed for ROI:  271
Reached
Reached
Stats computed for ROI:  272
Stats computed for ROI:  265
Reached
Reached
Stats computed for ROI:  273
Stats computed for ROI:  266
Reached
Stats computed for ROI:  267
Reached
Stats computed for ROI:  268
Reached
Stats computed for ROI:  269
CPU times: user 1.18 s, sys: 620 ms, total: 1.8 s
Wall time: 4.31 s

In [13]:
roi_voxel_stats_matrix = np.asarray(roi_voxel_stats_matrix)

In [15]:
autism_mean_path = opj(save_destination,'roi_voxel_stats_matrix')

np.savetxt(autism_mean_path,roi_voxel_stats_matrix,delimiter=',',header='min_pval,min_qval,p_lt_point_1,p_lt_point_01,p_lt_point_05,q_lt_point_1,q_lt_point_01,q_lt_point_05')

Calculating the number of voxels per ROI


In [ ]:
path_cwd = os.getcwd()
path_split_list = path_cwd.split('/')
s = path_split_list[0:-1]
s = opj('/',*s) # *s converts list to path, # also it is very important to add '/' in the begining so it is read as directory later

# Put in the path of atlas you wish to use
atlasPath = opj(s,'atlas/Full_brain_atlas_thr0-2mm/fullbrain_atlas_thr0-2mm.nii.gz')

# Read the atlas
atlasObject = nib.load(atlasPath)
atlas = atlasObject.get_data()

In [ ]:
x_dim, y_dim, z_dim = atlas.shape

# Keep track of number of voxels per ROI as well by using an array - num_voxels_in_ROI[]

num_voxels_in_ROI = np.zeros((num_ROIs,1)) # A column arrray containing number of voxels in each ROI

    
for i in range(x_dim):
    for j in range(y_dim):
        for k in range(z_dim):
            label = int(atlas[i,j,k]) - 1
            if label != -1:
                num_voxels_in_ROI[label,0] = num_voxels_in_ROI[label,0] + 1

In [ ]:
autism_mean_path = opj(save_destination,'num_voxels_in_ROI.txt')

np.savetxt(autism_mean_path,num_voxels_in_ROI,delimiter=',',header='Number of Voxels')

In [ ]:
# END---------------------------------------------------------------------------------------------

In [ ]:
# temp = np.random.rand(10,10)

In [ ]:
# np.shape(np.where(temp>0.5))[1]

In [ ]:
# temp

In [ ]:
# temp[0,:] = [1,2,3,4,5,6,7,8,9,10]

In [ ]:
# temp

In [ ]:
# a,b = (1,2)
# b

In [ ]: