Splitting a simulation

Included in this notebook:

Split a full simulation file into trajectories and the rest



In [1]:

    
%matplotlib inline
import matplotlib.pyplot as plt
import openpathsampling as paths
import numpy as np

The optimum way to use storage depends on whether you're doing production or analysis. For analysis, you should open the file as an AnalysisStorage object. This makes the analysis much faster.



In [2]:

    
%%time
storage = paths.AnalysisStorage("mstis.nc")









    



CPU times: user 8.86 s, sys: 359 ms, total: 9.22 s
Wall time: 9.53 s



In [3]:

    
st_split = paths.Storage('mstis_strip.nc', 'w')



In [4]:

    
# st_traj = paths.Storage('mstis_traj.nc', 'w')
# st_data = paths.Storage('mstis_data.nc', 'w')



In [5]:

    
st_split.fallback = storage



In [5]:

    
# st_data.fallback = storage

Store all trajectories completely in the data file



In [6]:

    
# st_data.snapshots.save(storage.snapshots[0])
# st_traj.snapshots.save(storage.snapshots[0])









    Out[6]:





UUID('cb90664f-80cc-11e6-90fa-0000000098cb')

Add a single snapshot as a reference and create the appropriate stores



In [6]:

    
st_split.snapshots.save(storage.snapshots[0])









    Out[6]:





UUID('cb90664f-80cc-11e6-90fa-0000000098cb')

Store only shallow trajectories (empty snapshots) in the main file

fix CVs first, rest is fine



In [7]:

    
cvs = storage.cvs



In [15]:

    
q = storage.snapshots.all()

fill weak cache from stored cache. This should be fast and we can later use the weak cache (as long as q exists) to fill the cache of the data file.



In [16]:

    
%%time
_ = [cv(q) for cv in cvs]









    



CPU times: user 1.94 s, sys: 41.1 ms, total: 1.98 s
Wall time: 1.96 s

Now that we have cached the CV values we can save the CVs in the new store. This will also set the disk cache to the new file and since the file is new this one is empty.



In [17]:

    
%%time
# this will also switch the storage cache to the new file
_ = map(st_split.cvs.save, storage.cvs)









    



CPU times: user 83.1 ms, sys: 289 ms, total: 372 ms
Wall time: 729 ms



In [9]:

    
# %%time
# # this will also switch the storage cache to the new file
# _ = map(st_data.cvs.save, storage.cvs)









    



CPU times: user 79.5 ms, sys: 57.7 ms, total: 137 ms
Wall time: 136 ms

if all cvs are really cached we can store snapshots now and the auto-complete will fill the CV disk store automatically when snapshots are saved. This takes a little while.



In [18]:

    
len(st_split.snapshots)









    Out[18]:





2



In [20]:

    
%%time
_ = map(st_split.trajectories.mention, storage.trajectories)









    



CPU times: user 37.2 s, sys: 688 ms, total: 37.9 s
Wall time: 38 s



In [ ]:

    
print len(st_split.snapshotspshots)



In [10]:

    
# %%time
# _ = map(st_data.trajectories.mention, storage.trajectories)









    



CPU times: user 32.8 s, sys: 410 ms, total: 33.3 s
Wall time: 33.3 s

Fill trajectory store only with trajectories and their snapshots. We are using lots of small snapshots and these are slow in comparison to large ones. So this will also take a minute or so.



In [11]:

    
%%time
_ = map(st_traj.trajectories.save, storage.trajectories)









    



CPU times: user 1min 58s, sys: 923 ms, total: 1min 59s
Wall time: 1min 59s

Finally try storing all steps from the simulation. This should contain ALL you need.



In [12]:

    
%%time
_ = map(st_data.steps.save, storage.steps)









    



CPU times: user 7.08 s, sys: 104 ms, total: 7.19 s
Wall time: 7.19 s

And compare file sizes



In [13]:

    
print 'Original file:', storage.file_size_str
print 'Data file:', st_data.file_size_str
print 'Traj file:', st_traj.file_size_str









    



Original file: 61.51MB
Data file: 49.50MB
Traj file: 25.74MB



In [25]:

    
print 'So we saved about %2.0f %%' % ((1.0 - st_data.file_size / float(storage.file_size)) * 100.0)









    



So we saved about 20 %

now we do the trick and use the small data file instead of the full simulation and see if that works.



In [26]:

    
st_data.close()
st_traj.close()
storage.close()



In [ ]:

    
st_data.snapshots.only_mention = True