scheme:

  • 1) for data transfer, pick 1st sleep api (h2d) fo stream-0, current cc = 1 (concurrency),
  • 2) check whether there is overalp with stream-
  • 2) if there is overlap, finish cc=1, start from cc++ (cc=2), predit the future ending time
  • 3) during the predicted ending time, check whether there is overlap with stream-2
  • 4) if there is overalap, finish cc=2, start from cc++ (cc=3), predict the future ending time
  • 5) go to step 3) , search through all the cuda streams
  • 6) for each time range, we need to find out how many apis have overlap and which-pair have conflicts or not

In [1]:
%load_ext autoreload
%autoreload 2

import warnings
import pandas as pd
import numpy as np
import os
import sys # error msg, add the modules
import operator # sorting
from math import *
import matplotlib.pyplot as plt

sys.path.append('../../')

import cuda_timeline
import read_trace
import avgblk
import cke
from model_param import *

warnings.filterwarnings("ignore", category=np.VisibleDeprecationWarning)

gpu info


In [2]:
gtx950 = DeviceInfo()
gtx950.sm_num = 6
gtx950.sharedmem_per_sm = 49152
gtx950.reg_per_sm = 65536
gtx950.maxthreads_per_sm = 2048

sm resource list and sm trace list


In [3]:
# init SM resources
SM_resList, SM_traceList = init_gpu(gtx950)

In [4]:
SM_resList[0]


Out[4]:
<model_param.SM_Stat instance at 0x7fee76eeecb0>

In [5]:
SM_traceList[0]


Out[5]:
sm_id block_id block_start block_end batch_id kernel_id active

Understand the input


In [6]:
trace_s1 = 'trace_s1_5m.csv'
df_trace_s1 = read_trace.Trace2dataframe(trace_s1)

trace_s3 = 'trace_s3_5m.csv'
df_trace_s3 = read_trace.Trace2dataframe(trace_s3)

In [7]:
#df_trace_s1

In [8]:
cuda_timeline.plot_trace(df_trace_s1)


/home/leiming/anaconda2/lib/python2.7/site-packages/matplotlib/axes/_base.py:1215: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if aspect == 'normal':
/home/leiming/anaconda2/lib/python2.7/site-packages/matplotlib/axes/_base.py:1220: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aspect in ('equal', 'auto'):

In [9]:
#cuda_timeline.plot_trace(df_trace_s2)

In [10]:
cuda_timeline.plot_trace(df_trace_s3)



In [11]:
df_3stream = read_trace.Get_timing_from_trace(df_trace_s3)
tot_runtime = read_trace.GetTotalRuntime(df_3stream)
print tot_runtime


53.626318

Kernel Info from the single stream


In [12]:
# extract kernel info from trace
# warning: currently lmted to one kernel

kernel = read_trace.GetKernelInfo(df_trace_s1, gtx950)

Dump_kernel_info(kernel)


Kernel Info
		blockDim 256.0
		gridkDim 19532.0
		regs 28.0
		shared memory 0.0
		runtime (ms) 11.914429
		average block execution time (ms) 0.0292737813268
		start time (ms) 0

set up cuda stream kernels


In [13]:
# for each stream, have a dd for each kernel
stream_kernel_list = []

stream_num = 3

for sid in range(stream_num):
    #print sid
    # key will be the kernel order
    # value will be the kernel info
    kern_dd = {}
    kern_dd[0] = Copy_kernel_info(kernel)
    stream_kernel_list.append(kern_dd)
    
    
Dump_kernel_info(stream_kernel_list[0][0])


Kernel Info
		blockDim 256.0
		gridkDim 19532.0
		regs 28.0
		shared memory 0.0
		runtime (ms) 11.914429
		average block execution time (ms) 0.0292737813268
		start time (ms) 0

start kernel from beginning


In [14]:
df_s1_trace_timing = read_trace.Get_timing_from_trace(df_trace_s1)
df_s1 = read_trace.Reset_starting(df_s1_trace_timing)

In [15]:
df_s1


Out[15]:
stream api_type start end size duration
0 0.0 h2d 0.000000 3.187298 19073.486 3.187298
1 0.0 h2d 3.188514 6.384227 19073.486 3.195713
2 0.0 kern 6.391203 18.305632 0.000 11.914429
3 0.0 d2h 18.311008 21.445857 19073.486 3.134849

set the h2d start for all the cuda streams


In [16]:
# find when to start the stream and update the starting pos for the trace
H2D_H2D_OVLP_TH = 3.158431

df_cke_list = cke.init_trace_list(df_s1, stream_num = stream_num, h2d_ovlp_th = H2D_H2D_OVLP_TH)


stream_startTime : 3.160431
stream_startTime : 6.320862

In [17]:
#df_cke_list[0]

In [18]:
#df_cke_list[1]

In [19]:
#df_cke_list[2]

merge all the cuda stream trace together


In [20]:
df_all_api = cke.init_sort_api_with_extra_cols(df_cke_list)

In [21]:
df_all_api


Out[21]:
start end api_type size_kb stream_id status bw kern_id bytes_done bytes_left current_pos pred_end
0 0.000000 3.187298 h2d 19073.486 0.0 sleep 5984.217980 None 0.0 19073.486 0.0 0.0
4 3.160431 6.347729 h2d 19073.486 1.0 sleep 5984.217980 None 0.0 19073.486 0.0 0.0
1 3.188514 6.384227 h2d 19073.486 0.0 sleep 5968.460247 None 0.0 19073.486 0.0 0.0
8 6.320862 9.508160 h2d 19073.486 2.0 sleep 5984.217980 None 0.0 19073.486 0.0 0.0
5 6.348945 9.544658 h2d 19073.486 1.0 sleep 5968.460247 None 0.0 19073.486 0.0 0.0
2 6.391203 18.305632 kern 0.000 0.0 sleep 0.000000 0 0.0 0.000 0.0 0.0
9 9.509376 12.705089 h2d 19073.486 2.0 sleep 5968.460247 None 0.0 19073.486 0.0 0.0
6 9.551634 21.466063 kern 0.000 1.0 sleep 0.000000 1 0.0 0.000 0.0 0.0
10 12.712065 24.626494 kern 0.000 2.0 sleep 0.000000 2 0.0 0.000 0.0 0.0
3 18.311008 21.445857 d2h 19073.486 0.0 sleep 6084.339628 None 0.0 19073.486 0.0 0.0
7 21.471439 24.606288 d2h 19073.486 1.0 sleep 6084.339628 None 0.0 19073.486 0.0 0.0
11 24.631870 27.766719 d2h 19073.486 2.0 sleep 6084.339628 None 0.0 19073.486 0.0 0.0

start algorithm


In [22]:
simPos = 0.0
more_streams = stream_num - 1

active_stream_dd = {}
for s in range(stream_num):
    active_stream_dd[s] = None

Dump_dd(active_stream_dd)


round_count = 1
while not cke.AllDone(df_all_api):
    
    #
    # check whether there is any call in sleep
    # if not, which means all wake(or done), finish them all and break the while loop
    if NoMoreSleepCalls(df_all_api):
        print('no more sleep calls')
        df_all_api = FinishRestWakeCalls(df_all_api)
        break
        
        
        
    #if round_count == 11: break
        
    df_all_api, r1, r1_stream = cke.pick_base_call(df_all_api)
    #
    # wake it up if r1 is in sleep
    if GetInfo(df_all_api, r1, 'status') == 'sleep': df_all_api = SetWake(df_all_api, r1)
    #
    # if r1 no in the pool add it
    if active_stream_dd[r1_stream] == None: active_stream_dd[r1_stream] = r1
    print('\n------------\n\n new round ({}) => row {}, stream-id {}'.format(round_count, r1, r1_stream))
    
    Dump_dd(active_stream_dd)
    
    
    print('simPos {}'.format(simPos))
    
    #if round_count == 11: break
    
    #
    # simulation position: simPos should be ahead of r1_start for wake call
    #r1_start = GetInfo(df_all_api, r1, 'start')
    #if r1_start >= simPos: simPos = r1_start
    #print('simulation position : {}'.format(simPos))
    
    #if round_count == 2: break
    
    
    
    
    
    prev_row = r1
    
    for i in range(0, more_streams):
        
        #if round_count == 4 and i == 0: break
        
        df_all_api, r2, r2_stream = cke.start_next_call(df_all_api, prev_row)
        print('=> pick row {}, stream-id {}'.format(r2, r2_stream))
        
        #if round_count == 10 and i == 1: break
            
        #
        # check r2 is already running, if yes,continue to pick next
        if active_stream_dd[r2_stream] == r2: 
            print('=> picked row {} is already running. select next'.format(r2))
            prev_row = r2
            #if round_count == 8 and i == 0: break
            continue
            
        #if round_count == 3 and i == 0: break
        #if round_count == 2: break
        
        
        # current position
        curPos = GetInfo(df_all_api, r2, 'start')
        print('current position : {}'.format(curPos))
        
        Dump_dd(active_stream_dd)
        
        #if round_count == 10 and i == 1: break    
        
        #if round == 2 and i == 0: break

        
        if active_stream_dd[r2_stream] == None:
            active_stream_dd[r2_stream] = r2        
            # update trace during the range
            print('simPos {}  curPos {}'.format(simPos, curPos))
            
            df_all_api,SM_resList, SM_traceList = cke.update_by_range(df_all_api, simPos, curPos,
                                                                      gtx950,
                                                                      SM_resList, SM_traceList,
                                                                      stream_kernel_list)
            #
            # find the kernel execution time from the sm trace table
            #result_kernel_runtime_dd = avgblk.Get_KernTime(SM_traceList)
            #print result_kernel_runtime_dd

            Dump_dd(active_stream_dd)
            
            #print GetInfo(df_all_api, 2, 'pred_end')
            #print GetInfo(df_all_api, 6, 'pred_end')
            
            
            #if round_count == 8 and i == 1: break
                             
        else:
            #
            # there is stream api ahead, find out which call and terminate it
            row_2nd = Find_prevapi_samestream(df_all_api, r2, r2_stream)
            print('end prev api call at row {}'.format(row_2nd))
            
            #if round_count == 10 and i == 1: break
                
                
            #
            # end the target row, update the bytes for other call
            df_all_api = cke.end_target_row(df_all_api, row_2nd, simPos, curPos)
            
            #if round_count == 9 and i == 1: break
                
            #
            # update curPos, since r2 start has been shifted right
            curPos = GetInfo(df_all_api, r2, 'start')
            df_all_api = UpdateCell(df_all_api, r2, 'current_pos', curPos)
            print('row {}, Updated current position : {}'.format(r2, curPos))
            
            #if round_count == 9 and i == 1: break
                
            #
            # assume there is no ovlp between row_2nd end and r2_start
            row2nd_end = GetInfo(df_all_api, row_2nd, 'end')
            df_all_api = cke.move_wake_for_coming_call(df_all_api, row2nd_end, curPos)
            #
            # update the count for active stream pool: remove row2nd, add r2 to the pool
            active_stream_dd[r2_stream] = r2
            #Dump_dd(active_stream_dd)
            
            #if round_count == 9 and i == 1: break
            
        #
        # shift right
        simPos = curPos
        #
        # update prev_row
        prev_row = r2
        
        #
        # dump active dd
        #Dump_dd(active_stream_dd)
        
        #if i == 0: break
        #if round_count == 6 and i == 1: break
    
        
    #
    # end of For loop
    # dump active dd
    Dump_dd(active_stream_dd)
    print('simPos {}  curPos {}'.format(simPos, curPos))
    
    
    #if round_count == 11: break
    
    
    #
    #
    # check whether the active pool is full, if yes, terminate the api that ends soon
    df_all_api, active_stream_dd, simPos = cke.check_activestream_and_update(df_all_api, active_stream_dd, simPos)
    
    print active_stream_dd
    
    #if round_count == 11: break
        
    round_count += 1


stream 0 : active_api None
stream 1 : active_api None
stream 2 : active_api None

------------

 new round (1) => row 0, stream-id 0.0
stream 0 : active_api 0
stream 1 : active_api None
stream 2 : active_api None
simPos 0.0
=> pick row 4, stream-id 1.0
current position : 3.160431
stream 0 : active_api 0
stream 1 : active_api None
stream 2 : active_api None
simPos 0.0  curPos 3.160431
wake list [0] 
h2d_list : [0]
d2h_list : []
kern_list : []
cc 1 for all the wake h2d list
stream 0 : active_api 0
stream 1 : active_api 4
stream 2 : active_api None
=> pick row 1, stream-id 0.0
current position : 3.188514
stream 0 : active_api 0
stream 1 : active_api 4
stream 2 : active_api None
end prev api call at row 0
simT 3.160431,  curT3.188514 
Before time 3.188514, wake list [0, 4] 
h2d cc 2
row 1, Updated current position : 3.215381
move_wake_for_coming_call, wake list [4] 
wake row 4, pred_end 6.374596
stream 0 : active_api 1
stream 1 : active_api 4
stream 2 : active_api None
simPos 3.215381  curPos 3.215381
{0: 1, 1: 4, 2: None}

------------

 new round (2) => row 4, stream-id 1.0
stream 0 : active_api 1
stream 1 : active_api 4
stream 2 : active_api None
simPos 3.215381
=> pick row 1, stream-id 0.0
=> picked row 1 is already running. select next
=> pick row 8, stream-id 2.0
current position : 6.320862
stream 0 : active_api 1
stream 1 : active_api 4
stream 2 : active_api None
simPos 3.215381  curPos 6.320862
wake list [4, 1] 
h2d_list : [4, 1]
d2h_list : []
kern_list : []
cc 2 for all the wake h2d list
stream 0 : active_api 1
stream 1 : active_api 4
stream 2 : active_api 8
stream 0 : active_api 1
stream 1 : active_api 4
stream 2 : active_api 8
simPos 6.320862  curPos 6.320862
check_activestream_and_update : wakelist [4, 1, 8]
row to end : 4, its stream 1.0
row_afterprevcall 5
-----
next call after : 5
simT 6.320862,  curT6.348945 
Before time 6.348945, wake list [4, 1, 8] 
h2d cc 3
currpos 11.1402855 to next call start 11.1415015
move_wake_for_coming_call, wake list [1, 8] 
wake row 1, pred_end 11.1767835
wake row 8, pred_end 12.721109
{0: 1, 1: None, 2: 8}

------------

 new round (3) => row 1, stream-id 0.0
stream 0 : active_api 1
stream 1 : active_api None
stream 2 : active_api 8
simPos 11.1415015
=> pick row 8, stream-id 2.0
=> picked row 8 is already running. select next
=> pick row 5, stream-id 1.0
current position : 11.1415015
stream 0 : active_api 1
stream 1 : active_api None
stream 2 : active_api 8
simPos 11.1415015  curPos 11.1415015
wake list [1, 8, 5] 
h2d_list : [1, 8, 5]
d2h_list : []
kern_list : []
cc 3 for all the wake h2d list
stream 0 : active_api 1
stream 1 : active_api 5
stream 2 : active_api 8
stream 0 : active_api 1
stream 1 : active_api 5
stream 2 : active_api 8
simPos 11.1415015  curPos 11.1415015
check_activestream_and_update : wakelist [1, 8, 5]
row to end : 1, its stream 0.0
row_afterprevcall 2
-----
next call after : 2
simT 11.1415015,  curT6.41807 
Before time 6.41807, wake list [1, 8] 
h2d cc 2
currpos 11.2120655 to next call start 11.2190415
move_wake_for_coming_call, wake list [8, 5] 
wake row 8, pred_end 12.756391
wake row 5, pred_end 14.3372145
{0: None, 1: 5, 2: 8}

------------

 new round (4) => row 8, stream-id 2.0
stream 0 : active_api None
stream 1 : active_api 5
stream 2 : active_api 8
simPos 11.2190415
=> pick row 5, stream-id 1.0
=> picked row 5 is already running. select next
=> pick row 2, stream-id 0.0
current position : 11.2190415
stream 0 : active_api None
stream 1 : active_api 5
stream 2 : active_api 8
simPos 11.2190415  curPos 11.2190415
wake list [8, 5, 2] 
h2d_list : [8, 5]
d2h_list : []
kern_list : [2]
cc 2 for all the wake h2d list
sorted kernel rows: [2]
find kernel ? False : row 2, in SM trace
kid = 0
Kernel Info
		blockDim 256.0
		gridkDim 19532.0
		regs 28.0
		shared memory 0.0
		runtime (ms) 11.914429
		average block execution time (ms) 0.0292737813268
		start time (ms) 11.2190415
{0.0: [11.219041500000255, 23.133470500000527]}
updating kernel row2, with pred_start 11.2190415, pred_end 23.1334705
kernel model still need some work!
stream 0 : active_api 2
stream 1 : active_api 5
stream 2 : active_api 8
stream 0 : active_api 2
stream 1 : active_api 5
stream 2 : active_api 8
simPos 11.2190415  curPos 11.2190415
check_activestream_and_update : wakelist [8, 5, 2]
row to end : 8, its stream 2.0
row_afterprevcall 9
-----
next call after : 9
simT 11.2190415,  curT9.509376 
Before time 9.509376, wake list [8] 
h2d cc 1
currpos 12.756391 to next call start 12.757607
move_wake_for_coming_call, wake list [5, 2] 
wake row 5, pred_end 14.3372145
wake row 2, pred_end 23.1334705
{0: 2, 1: 5, 2: None}

------------

 new round (5) => row 5, stream-id 1.0
stream 0 : active_api 2
stream 1 : active_api 5
stream 2 : active_api None
simPos 12.757607
=> pick row 2, stream-id 0.0
=> picked row 2 is already running. select next
=> pick row 9, stream-id 2.0
current position : 12.757607
stream 0 : active_api 2
stream 1 : active_api 5
stream 2 : active_api None
simPos 12.757607  curPos 12.757607
wake list [5, 2, 9] 
h2d_list : [5, 9]
d2h_list : []
kern_list : [2]
cc 2 for all the wake h2d list
sorted kernel rows: [2]
find kernel ? True : row 2, in SM trace
kid = 0
{0.0: [11.219041500000255, 23.133470500000527]}
updating kernel row2, with pred_start 11.2190415, pred_end 23.1334705
kernel model still need some work!
stream 0 : active_api 2
stream 1 : active_api 5
stream 2 : active_api 9
stream 0 : active_api 2
stream 1 : active_api 5
stream 2 : active_api 9
simPos 12.757607  curPos 12.757607
check_activestream_and_update : wakelist [5, 2, 9]
row to end : 5, its stream 1.0
row_afterprevcall 6
-----
next call after : 6
simT 12.757607,  curT14.3441905 
Before time 14.3441905, wake list [5, 2, 9] 
h2d cc 2
currpos 19.132649 to next call start 19.139625
move_wake_for_coming_call, wake list [2, 9] 
wake row 2, pred_end 23.1334705
wake row 9, pred_end 19.140841
{0: 2, 1: None, 2: 9}

------------

 new round (6) => row 2, stream-id 0.0
stream 0 : active_api 2
stream 1 : active_api None
stream 2 : active_api 9
simPos 19.139625
=> pick row 9, stream-id 2.0
=> picked row 9 is already running. select next
=> pick row 6, stream-id 1.0
current position : 19.139625
stream 0 : active_api 2
stream 1 : active_api None
stream 2 : active_api 9
simPos 19.139625  curPos 19.139625
wake list [2, 9, 6] 
h2d_list : [9]
d2h_list : []
kern_list : [2, 6]
cc 1 for all the wake h2d list
sorted kernel rows: [2, 6]
find kernel ? True : row 2, in SM trace
kid = 0
find kernel ? False : row 6, in SM trace
kid = 1
Kernel Info
		blockDim 256.0
		gridkDim 19532.0
		regs 28.0
		shared memory 0.0
		runtime (ms) 11.914429
		average block execution time (ms) 0.0292737813268
		start time (ms) 19.139625
{0.0: [11.219041500000255, 23.133470500000527], 1.0: [23.104196718673744, 35.047899500000717]}
updating kernel row2, with pred_start 11.2190415, pred_end 23.1334705
updating kernel row6, with pred_start 23.1041967187, pred_end 35.0478995
kernel model still need some work!
stream 0 : active_api 2
stream 1 : active_api 6
stream 2 : active_api 9
stream 0 : active_api 2
stream 1 : active_api 6
stream 2 : active_api 9
simPos 19.139625  curPos 19.139625
check_activestream_and_update : wakelist [2, 9, 6]
row to end : 9, its stream 2.0
row_afterprevcall 10
-----
next call after : 10
simT 19.139625,  curT15.960296 
Before time 15.960296, wake list [2, 9] 
h2d cc 1
currpos 19.140841 to next call start 19.147817
move_wake_for_coming_call, wake list [2] 
wake row 2, pred_end 23.1334705
{0: 2, 1: 6, 2: None}

------------

 new round (7) => row 2, stream-id 0.0
stream 0 : active_api 2
stream 1 : active_api 6
stream 2 : active_api None
simPos 19.147817
=> pick row 6, stream-id 1.0
=> picked row 6 is already running. select next
=> pick row 10, stream-id 2.0
current position : 19.147817
stream 0 : active_api 2
stream 1 : active_api 6
stream 2 : active_api None
simPos 19.147817  curPos 19.147817
wake list [2, 10] 
h2d_list : []
d2h_list : []
kern_list : [2, 10]
sorted kernel rows: [2, 10]
find kernel ? True : row 2, in SM trace
kid = 0
find kernel ? False : row 10, in SM trace
kid = 2
Kernel Info
		blockDim 256.0
		gridkDim 19532.0
		regs 28.0
		shared memory 0.0
		runtime (ms) 11.914429
		average block execution time (ms) 0.0292737813268
		start time (ms) 19.147817
{0.0: [11.219041500000255, 23.133470500000527], 1.0: [23.104196718673744, 35.047899500000717], 2.0: [35.018625718673938, 46.962328499999835]}
updating kernel row2, with pred_start 11.2190415, pred_end 23.1334705
updating kernel row10, with pred_start 35.0186257187, pred_end 46.9623285
kernel model still need some work!
stream 0 : active_api 2
stream 1 : active_api 6
stream 2 : active_api 10
stream 0 : active_api 2
stream 1 : active_api 6
stream 2 : active_api 10
simPos 19.147817  curPos 19.147817
check_activestream_and_update : wakelist [2, 6, 10]
row to end : 2, its stream 0.0
row_afterprevcall 3
-----
next call after : 3
simT 19.147817,  curT23.1388465 
Before time 23.1388465, wake list [2, 6] 
currpos 23.1334705 to next call start 23.1388465
move_wake_for_coming_call, wake list [6] 
wake row 6, pred_end 35.0478995
{0: None, 1: 6, 2: 10}

------------

 new round (8) => row 6, stream-id 1.0
stream 0 : active_api None
stream 1 : active_api 6
stream 2 : active_api 10
simPos 23.1388465
=> pick row 10, stream-id 2.0
=> picked row 10 is already running. select next
=> pick row 3, stream-id 0.0
current position : 23.1388465
stream 0 : active_api None
stream 1 : active_api 6
stream 2 : active_api 10
simPos 23.1388465  curPos 23.1388465
wake list [6, 3] 
h2d_list : []
d2h_list : [3]
kern_list : [6]
cc 1 for all the wake d2h list
sorted kernel rows: [6]
find kernel ? True : row 6, in SM trace
kid = 1
{0.0: [11.219041500000255, 23.133470500000527], 1.0: [23.104196718673744, 35.047899500000717], 2.0: [35.018625718673938, 46.962328499999835]}
updating kernel row6, with pred_start 23.1041967187, pred_end 35.0478995
kernel model still need some work!
stream 0 : active_api 3
stream 1 : active_api 6
stream 2 : active_api 10
stream 0 : active_api 3
stream 1 : active_api 6
stream 2 : active_api 10
simPos 23.1388465  curPos 23.1388465
check_activestream_and_update : wakelist [6, 10, 3]
row to end : 3, its stream 0.0
row_afterprevcall None
-----
{0: None, 1: 6, 2: 10}

------------

 new round (9) => row 6, stream-id 1.0
stream 0 : active_api None
stream 1 : active_api 6
stream 2 : active_api 10
simPos 26.2736955
=> pick row 10, stream-id 2.0
=> picked row 10 is already running. select next
=> pick row 7, stream-id 1.0
current position : 35.0532755
stream 0 : active_api None
stream 1 : active_api 6
stream 2 : active_api 10
end prev api call at row 6
simT 26.2736955,  curT35.0532755 
Before time 35.0532755, wake list [6, 10] 
row 7, Updated current position : 35.0532755
move_wake_for_coming_call, wake list [10] 
wake row 10, pred_end 46.9623285
stream 0 : active_api None
stream 1 : active_api 7
stream 2 : active_api 10
simPos 35.0532755  curPos 35.0532755
{0: None, 1: 7, 2: 10}

------------

 new round (10) => row 10, stream-id 2.0
stream 0 : active_api None
stream 1 : active_api 7
stream 2 : active_api 10
simPos 35.0532755
=> pick row 7, stream-id 1.0
=> picked row 7 is already running. select next
=> pick row 11, stream-id 2.0
current position : 46.9677045
stream 0 : active_api None
stream 1 : active_api 7
stream 2 : active_api 10
end prev api call at row 10
simT 35.0532755,  curT46.9677045 
Before time 46.9677045, wake list [10, 7] 
row 11, Updated current position : 46.9677045
move_wake_for_coming_call, wake list [7] 
wake row 7, pred_end 38.1881245
stream 0 : active_api None
stream 1 : active_api 7
stream 2 : active_api 11
simPos 46.9677045  curPos 46.9677045
{0: None, 1: 7, 2: 11}
no more sleep calls

In [23]:
df_all_api


Out[23]:
start end api_type size_kb stream_id status bw kern_id bytes_done bytes_left current_pos pred_end
0 0.000000 3.214165 h2d 19073.486 0.0 done 5984.217980 None 19073.486 0.0 3.214165 3.214165
4 3.160431 11.140286 h2d 19073.486 1.0 done 5984.217980 None 19073.486 0.0 11.140286 11.140286
1 3.215381 11.212066 h2d 19073.486 0.0 done 5968.460247 None 19073.486 0.0 11.212066 11.212066
8 6.320862 12.756391 h2d 19073.486 2.0 done 5984.217980 None 19073.486 0.0 12.756391 12.756391
5 11.141502 19.132649 h2d 19073.486 1.0 done 5968.460247 None 19073.486 0.0 19.132649 19.132649
2 11.219042 23.133471 kern 0.000 0.0 done 0.000000 0 0.000 0.0 23.133471 23.133471
9 12.757607 19.140841 h2d 19073.486 2.0 done 5968.460247 None 19073.486 0.0 19.140841 19.140841
6 23.104197 35.047900 kern 0.000 1.0 done 0.000000 1 0.000 0.0 35.047900 35.047900
10 35.018626 46.962328 kern 0.000 2.0 done 0.000000 2 0.000 0.0 46.962328 46.962328
3 23.138847 26.273696 d2h 19073.486 0.0 done 6084.339628 None 19073.486 0.0 26.273696 26.273696
7 35.053276 38.188125 d2h 19073.486 1.0 done 6084.339628 None 19073.486 0.0 38.188125 38.188125
11 46.967704 50.102553 d2h 19073.486 2.0 done 6084.339628 None 19073.486 0.0 50.102553 50.102553

In [24]:
#
# run above
#