Ch 02: Concept 09

Using Queues

If you have a lot of training data, you probably don't want to load it all into memory at once. The QueueRunner in TensorFlow is a tool to efficiently employ a queue data-structure in a multi-threaded way.


In [1]:
import tensorflow as tf
import numpy as np

We will be running multiple threads, so let's figure out the number of CPUs:


In [2]:
import multiprocessing
NUM_THREADS = multiprocessing.cpu_count()

Generate some fake data to work with:


In [3]:
xs = np.random.randn(100, 3)
ys = np.random.randint(0, 2, size=100)

Here's a couple concrete examples of our data:


In [4]:
xs_and_ys = zip(xs, ys)
for _ in range(5):
    x, y = next(xs_and_ys)
    print('Input {}  --->  Output {}'.format(x, y))


Input [ 1.46034759  0.71462742  0.73288402]  --->  Output 0
Input [ 1.1537654  -0.09128405  0.08036941]  --->  Output 1
Input [-0.61164559 -0.19188485  0.06064167]  --->  Output 0
Input [ 0.1007337   0.34815357  0.24346031]  --->  Output 0
Input [-1.25581117  1.44738085  1.15035257]  --->  Output 0

Define a queue:


In [5]:
queue = tf.FIFOQueue(capacity=1000, dtypes=[tf.float32, tf.int32])

Set up the enqueue and dequeue ops:


In [6]:
enqueue_op = queue.enqueue_many([xs, ys])
x_op, y_op = queue.dequeue()

Define a QueueRunner:


In [7]:
qr = tf.train.QueueRunner(queue, [enqueue_op] * 4)

Now that all variables and ops have been defined, let's get started with a session:


In [8]:
sess = tf.InteractiveSession()

Create threads for the QueueRunner:


In [9]:
coord = tf.train.Coordinator()
enqueue_threads = qr.create_threads(sess, coord=coord, start=True)

Test out dequeueing:


In [10]:
for _ in range(100):
    if coord.should_stop():
        break
    x, y = sess.run([x_op, y_op])
    print(x, y)
coord.request_stop()
coord.join(enqueue_threads)


[ 1.46034753  0.71462744  0.73288405] 0
[ 1.15376544 -0.09128405  0.08036941] 1
[-0.61164558 -0.19188486  0.06064167] 0
[ 0.1007337   0.34815356  0.24346031] 0
[-1.25581121  1.4473809   1.1503526 ] 0
[ 0.60369009 -0.87942719 -1.37121975] 1
[ 1.30641925  1.55316997  1.01789773] 0
[ 0.0575242   0.59463078  0.47600508] 1
[-1.22782397 -0.86792755  1.37459588] 1
[-0.27896652  0.51645088  1.36873603] 0
[-0.34542757  0.79360306  0.32000065] 0
[-0.46792462 -0.31817994  0.91739392] 0
[ 0.24787657  0.83848852  1.16125166] 0
[-0.46220389 -0.09412029 -0.9981451 ] 1
[ 0.06739734 -1.08405316 -0.3582162 ] 1
[-1.2644819  -0.27479929  1.15882337] 1
[-0.68015367 -0.10199564  1.4274267 ] 0
[-0.48884565 -0.39484504  0.1496018 ] 1
[ 1.48414564 -0.43943462 -0.12646018] 0
[ 0.49450573  0.42091215 -0.17693481] 0
[ 0.02265234  0.99832052  0.26808155] 1
[-0.94086462  1.67000341  0.92434174] 1
[-0.50961769 -0.39044595 -0.5737586 ] 0
[-0.95702702  0.61196166 -0.86487901] 1
[-0.6125344  -0.30916786 -1.06602347] 1
[-1.91383719  0.26860073  0.50380921] 1
[-0.14638679  0.11614402  1.36613548] 1
[-0.56817967  1.4221288   0.99365205] 0
[-0.04597072  0.43875724 -0.4809106 ] 0
[-0.2000681  -0.2384561   0.06599616] 0
[ 0.5862993   0.85386461  0.82285357] 1
[ 1.64371336 -0.46838599  0.22755136] 0
[ 0.21683638 -0.96399426  1.78278649] 1
[ 0.03778305  2.49208736  0.07467758] 0
[-1.48958826 -0.11699235  0.98281074] 1
[-0.27623582 -0.41658697 -0.89554274] 0
[-1.64742625  1.83507264 -0.76936585] 0
[-1.5386405   0.14272654  0.17047048] 1
[ 0.63654041  1.75451732 -1.14198494] 0
[-0.57061732  0.11121389  1.39394116] 1
[ 1.94736981 -0.36588097  0.54801333] 1
[-0.56976408 -1.36990237 -0.9922803 ] 1
[-2.47653961  1.19603479 -0.3038739 ] 0
[-0.76740891 -0.49611184  0.47167206] 0
[ 1.62004089  0.13268068  0.28845155] 0
[-0.91749012 -0.30151108 -0.08271972] 0
[-0.21053326 -0.16114895 -0.52424961] 1
[ 0.19968066  0.2387522   2.0314014 ] 0
[-0.29072183  0.53720349 -0.38972732] 0
[-0.85891634 -0.26684314 -1.91741192] 1
[-2.07077003  1.97488022 -0.92741841] 0
[ 2.37270904  2.19385314 -0.29643178] 0
[-0.18054648 -0.1651988   1.70858753] 1
[-0.27851281 -0.13095042  0.30613536] 1
[-0.13653868 -0.14431253  1.3018136 ] 1
[-1.79938364  0.26698261 -0.3283855 ] 0
[-0.43491617 -0.8737886  -0.48871836] 1
[-0.27275884  0.08004636 -0.34334385] 0
[-0.06538768 -0.47280514 -1.82918119] 0
[ 1.72329473  0.6359638   1.53474641] 0
[ 0.88200653  0.87051851  0.17676826] 1
[-2.22127795 -0.39812142  0.69118947] 0
[-0.90146214  0.23153968 -1.07890677] 0
[-0.66513097 -0.74897975 -1.9886812 ] 0
[ 0.95217085 -0.1361241  -0.81558466] 1
[ 0.97319698  0.10349847  1.78010297] 0
[ 0.54321396  1.10134006 -1.03641176] 1
[ 0.46445891  0.56387979  0.10383373] 0
[ 0.22231635 -1.20880091  0.20125042] 1
[ 0.56338882 -0.76195502 -0.33035895] 0
[ 0.13885871  0.62347603  0.32560909] 0
[-0.63413048  0.19185983  1.65251637] 1
[ 0.81965917 -0.14427175 -0.9943186 ] 0
[ 1.98786604 -1.38118052 -0.34296793] 0
[-0.49028778 -0.30242845  0.81718981] 0
[ 0.48434621 -1.3200016  -0.32307461] 0
[-0.91041267 -0.34315997  0.71205115] 0
[ 0.61457998 -0.85814965  0.6939835 ] 0
[-0.40195578 -1.11846507 -0.19713871] 1
[-0.47889531 -0.75685191  1.68955612] 1
[ 1.51117146 -2.23529124  1.13895822] 0
[-0.00831293 -0.50950557  0.08648733] 1
[-0.47011089  1.04781067 -0.05893843] 1
[-0.34855339 -0.5695411  -0.12196264] 1
[-0.47251806 -0.49479187  0.27609721] 0
[-2.04546118 -0.16185458  1.42348552] 0
[-0.67136103 -0.16650072  0.3609505 ] 0
[ 1.22566068  1.18665588 -1.87292075] 0
[-0.80474126 -0.1114784   0.00531922] 1
[ 0.62691861 -3.26328206 -0.39003551] 0
[-0.77470082 -1.23692167 -1.55790484] 0
[-0.49005547 -0.19645052 -0.21566501] 1
[-0.44095206 -0.13273652 -0.59810853] 0
[-0.9750855  -0.46043435  0.06064714] 1
[-0.181191   -0.12452056  0.23064452] 1
[-0.34818363 -1.13179028  1.20628965] 0
[-1.58196092 -1.3506341  -2.05767131] 1
[-1.66225421 -0.43541616  1.55258   ] 0
[-0.12949325 -0.15456693  0.04389611] 0
[ 0.24592777  0.11407969 -0.31221709] 1