Essentia.streaming Python tutorial

This demo will show how to use Essentia in streaming mode.

The main difference between standard and streaming is that the standard mode is imperative while the streaming mode is declarative. That means that in standard mode, you tell exactly the computer what to do, whereas in the streaming mode, you "declare" what is needed to be done, and you let the computer do it itself. One big advantage of the streaming mode is that the memory consumption is greatly reduced, as you don't need to load the entire audio in memory. Let's have a look at it.

As usual, first import the essentia module:



In [13]:

    
import essentia
from essentia.streaming import *

Instantiate our algorithms:



In [16]:

    
loader = MonoLoader(filename = '../../../test/audio/recorded/dubstep.wav')
frameCutter = FrameCutter(frameSize = 1024, hopSize = 512)
w = Windowing(type = 'hann')
spec = Spectrum()
mfcc = MFCC()

In streaming, instead of calling algorithms like functions, we need to connect their inputs and outputs. This is done using the >> operator.

For example, the graph we want to connect looks like this:

----------      ------------      -----------      --------------      --------------
MonoLoader      FrameCutter       Windowing        Spectrum            MFCC
     audio ---> signal frame ---> frame frame ---> frame spectrum ---> spectrum bands ---> ???
                                                                                mfcc  ---> ???
----------      ------------      -----------      --------------      --------------



In [17]:

    
loader.audio >> frameCutter.signal
frameCutter.frame >> w.frame >> spec.frame
spec.spectrum >> mfcc.spectrum









    Out[17]:





<essentia.streaming._StreamConnector instance at 0x73ebc20>

When building a network, all inputs need to be connected, no matter what, otherwise the network cannot be started.



In [18]:

    
essentia.run(loader)









    



---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-5a68facf7b1d> in <module>()
----> 1 essentia.run(loader)

/usr/local/lib/python2.7/dist-packages/essentia/__init__.py in run(gen)
    146     if isinstance(gen, VectorInput) and not gen.connections.values()[0]:
    147         raise EssentiaError('VectorInput is not connected to anything...')
--> 148     return _essentia.run(gen)
    149 
    150 log.debug(EPython, 'Successfully imported essentia python module (log fully available and synchronized with the C++ one)')

RuntimeError: MFCC::bands is not connected to any sink...

In our case, the outputs of the MFCC algorithm were not connected anywhere. Let's store mfcc values in the pool and ignore bands values.

----------      ------------      -----------      --------------      --------------
MonoLoader      FrameCutter       Windowing        Spectrum            MFCC
     audio ---> signal frame ---> frame frame ---> frame spectrum ---> spectrum bands ---> NOWHERE
                                                                                mfcc  ---> Pool: lowlevel.mfcc
----------      ------------      -----------      --------------      --------------



In [19]:

    
pool = essentia.Pool()

mfcc.bands >> None
mfcc.mfcc >> (pool, 'lowlevel.mfcc')

essentia.run(loader)

print 'Pool contains %d frames of MFCCs' % len(pool['lowlevel.mfcc'])









    



Pool contains 592 frames of MFCCs

Let's try writing directly to a text file, no pool and no yaml files

We first need to disconnect the old connection to the pool to avoid putting the same data in there again.



In [20]:

    
mfcc.mfcc.disconnect((pool, 'lowlevel.mfcc'))

We create a FileOutput and connect it. It is a special connection that has no input, because it can actually take any type of input (the other algorithms will complain if you try to connect an output to an input of a different type).



In [21]:

    
fileout = FileOutput(filename = 'mfccframes.txt')
mfcc.mfcc >> fileout









    Out[21]:





<essentia.streaming.StreamingAlgo at 0x73ec5f0>

Reset the network otherwise the loader in particular will not do anything useful, and rerun the network



In [22]:

    
essentia.reset(loader)
essentia.run(loader)

This is the resulting file:



In [ ]:

    
!less mfccframes.txt









    



[-430.671, 87.7917, -10.1204, -50.172, -17.9259, -36.4849, -17.5709, -5.72505, -16.6404, 8.64975, -7.41039, 5.7051, 7.18054]
[-490.824, 101.549, 68.3375, 10.5324, 9.86464, -21.2722, -12.467, -11.8749, -24.2667, -8.02748, -26.5459, -25.3716, -31.5998]
[-515.915, 90.4185, 54.5073, 25.2965, 18.2453, 1.56025, 10.0262, 21.2547, 2.83289, 7.16084, -25.8393, -22.4263, -29.8229]
[-526.075, 76.321, 33.0371, 15.6267, 16.1482, 1.94902, 26.5443, 40.805, 20.866, 20.7323, -16.962, -23.6936, -39.9292]
[-530.409, 62.8531, 17.8901, 17.2312, 19.4443, 6.44692, 35.9218, 37.0124, 9.91325, 30.9235, -10.691, -12.6595, -30.0003]
[-532.03, 66.9765, 15.174, 4.41039, 6.51185, 18.4618, 41.4819, 30.0178, 13.5438, 19.5735, -19.7553, -2.62841, -12.9201]
[-523.106, 85.9242, 15.2094, 11.4087, 9.95426, 19.4773, 20.8585, 27.0054, 19.3617, 19.016, -13.5927, -3.25358, -11.339]
[-532.996, 90.4333, 13.19, 8.79797, 20.2316, 15.791, 23.7306, 34.2449, 11.5618, 20.3763, -18.6916, -10.9794, -20.2573]
[-539.285, 74.0864, 20.9641, 18.1156, 11.1982, 6.72211, 25.9186, 38.2328, 8.60174, 16.578, -22.699, -19.8375, -27.6012]
[-512.555, 60.0025, 25.2891, 3.13255, 18.0855, -2.79687, 22.4047, 25.8552, 6.91857, 11.1513, -10.3943, -17.6128, -8.85415]
[-512.51, 57.8996, 30.54, -8.19113, 9.24241, -7.52199, 22.1372, 8.49042, 5.19367, 21.0422, -23.5516, -11.4846, -1.84778]
[-526.319, 65.6214, 20.5947, -1.63075, 5.35529, -11.1861, 3.16074, 31.1543, 15.7641, 24.6126, -16.7499, -5.88053, -17.8956]
[-539.224, 76.0849, 9.72453, -18.3865, 4.24167, -8.58866, 4.90287, 16.0067, 12.5663, 10.3253, -24.8418, -10.6817, -28.5775]
[-539.632, 65.1406, 14.7446, -9.75933, 3.81453, -1.8996, 14.3402, 21.5438, -0.942257, 9.80541, -34.0672, -20.4509, -30.8719]
[-530.661, 66.9916, 9.81611, -8.764, 7.43533, -4.98392, 11.8894, 23.2671, 6.46644, 18.2877, -11.8885, -6.97351, -17.6302]
[-535.676, 93.0305, 3.53535, -1.2581, 4.97832, 0.448381, 15.6231, 19.0904, 6.76488, 11.6231, -14.1466, 3.37447, -6.67421]
[-542.822, 77.1144, 6.2329, -5.24113, -4.40924, -2.91157, 15.8702, 26.7028, 12.0681, 16.2545, -5.97315, 5.64465, -11.0655]
[-538.625, 80.4892, 15.728, 0.430267, 1.37007, 0.846769, 12.069, 33.6801, 24.347, 18.6202, -12.545, -0.618216, -24.0178]
[-543.993, 75.0337, 2.45142, -3.08125, 6.12705, 7.48922, 20.7205, 34.5464, 14.0901, 15.1855, -15.196, -5.7751, -16.9295]
[-546.083, 39.7155, 28.0355, -7.47366, -0.486553, -6.15708, 24.7682, 32.6375, 24.4993, -2.43192, -10.1961, -11.0282, -11.8204]
[-522.732, 56.4225, 14.9481, -8.60417, 11.4418, -0.335148, 30.969, 24.1089, 22.346, 2.62507, -7.51814, -10.4893, -14.2436]
[-537.597, 46.9045, 1.08502, -7.52336, 10.8001, 3.71163, 40.0161, 15.1915, 14.8837, 7.42619, -10.9246, -15.7762, -12.8223]
[-545.106, 60.3563, 19.1322, -5.73659, 8.88274, -5.72114, 21.0186, 23.2324, 27.4492, 14.3531, -8.33187, -25.0716, -11.0766]
[-535.682, 60.2103, 17.8433, -5.057, 16.0371, -5.26712, 19.4347, 25.6891, 27.7371, 14.765, -10.9396, -16.4914, -9.45567]
[-545.443, 66.0985, -0.351505, -4.25406, 13.5872, 9.13861, 24.9208, 22.3003, 6.57347, -8.15397, -15.7383, -24.7124, -7.04774]
[-550.263, 88.9407, 7.59371, 13.1086, 1.73592, -5.69968, 12.7584, 21.7721, 19.0988, 7.48752, -7.27921, -29.9144, -12.9477]
[-554.875, 90.3536, -21.1439, 19.8487, 6.22874, 13.8184, 22.6209, 13.8373, 7.39717, 0.177366, -13.958, -21.9158, -7.70014]
[-546.982, 87.7155, -4.66905, 15.0691, 7.76933, 18.9313, 15.733, 10.617, 4.98855, -1.89248, -19.197, -22.1387, -16.5219]
[-527.187, 35.3692, 30.0158, -18.2529, 25.0347, 11.5841, 10.1781, 16.3096, 7.23779, 14.4277, -6.1009, -12.9349, -9.33879]
[-526.383, 65.1199, 17.6716, -13.3181, 13.378, 0.943882, 17.7097, 18.7745, -3.74555, 15.2577, -12.2919, -1.12687, -24.3105]
[-531.575, 76.2362, 11.9965, -6.61338, 9.75403, 0.190699, 11.1553, 13.5266, 4.92063, 13.6786, -14.4588, -6.66382, -25.9016]
[-539.532, 70.9667, 12.5404, -13.571, 12.6431, 11.8312, 22.1622, 19.7132, 5.65562, 18.5885, -9.71125, -2.66445, -32.761]
[-536.862, 61.65, 10.3666, -11.3195, 6.05235, -5.77159, 0.634165, 16.0269, 6.89768, 19.4605, -27.7461, -5.69038, -21.2543]
[-539.631, 81.382, 0.726603, -5.67793, 7.97709, 2.79132, 15.1069, 23.4298, 5.65112, 15.3011, -14.8289, -4.23787, -9.53112]
[-551.754, 74.654, -0.438913, -5.14595, 2.58914, 2.518, 13.053, 19.0255, 4.83003, 21.3977, -10.9551, -7.29639, -2.66554]
[-542.261, 79.5445, 3.55489, -3.74235, -0.446783, -5.52258, 9.68019, 20.6621, 11.6928, 13.635, -17.6341, -3.40418, -22.1211]
[-546.893, 70.3711, -10.0184, -16.2791, 0.182053, 8.86511, 21.3682, 32.0758, 10.3304, 25.6391, -4.60118, 2.09308, -28.1453]
[-540.669, 54.8316, 2.44856, -24.9267, 9.10489, -6.42201, 17.988, 19.7855, 13.2895, 5.29297, -7.84599, -13.9375, -17.7407]
[-542.237, 41.594, 1.06617, -20.8597, 10.7887, -3.45465, 18.743, 23.2737, 25.8101, 13.9783, -8.49092, -10.3478, -6.88474]
[-558.651, 39.4724, 0.235502, -10.0888, 28.4602, 14.6525, 34.1478, 21.7554, 12.0407, -5.94662, -22.8585, -28.6246, -5.93121]
[-568.023, 21.6655, -7.85221, -27.3913, -2.80562, -2.79098, 31.294, 34.5361, 30.607, 1.72647, -10.9985, -13.8208, -6.30239]
[-575.76, 10.9422, -5.98344, -21.1316, 5.73071, 7.95686, 31.9707, 34.4075, 20.4644, 4.57219, -12.3683, -19.2094, -15.4981]
[-594.335, 5.81063, -9.25603, -16.764, 5.61543, 8.21954, 35.175, 37.6078, 25.5661, 3.36957, -9.78766, -23.612, -17.6024]
[-604.968, 16.0413, -16.7058, -10.7215, -11.7695, 0.271067, 28.5352, 39.8839, 27.3278, -4.05683, -23.1454, -22.4309, -18.8693]
[-593.271, 32.7477, -16.4536, 2.27639, -7.05227, 0.331869, 19.0637, 31.0591, 23.9327, 5.80595, -23.0291, -24.2519, -13.8808]
[-596.032, 48.0299, -11.3058, 6.49618, -1.66786, 7.89659, 22.426, 37.753, 24.8686, 1.32107, -20.208, -23.0684, -20.7624]
[-563.358, 6.28607, 9.73108, -22.7878, -2.58262, -4.71011, 18.5552, 38.8026, 19.2201, 0.264616, -22.9942, -17.7035, -21.9024]
[-548.034, 26.0329, 20.5361, -22.5857, 7.88896, -1.83822, 19.149, 27.7694, 11.4569, 4.42627, -17.5259, -16.6323, -30.8919]
[-552.628, 45.1074, 8.1347, -16.9592, -0.514574, 0.262861, 21.3038, 24.9201, 9.08614, 21.7813, -13.4527, -8.82629, -26.8971]
[-547.505, 56.5691, 2.20271, -15.6015, -6.00866, -12.8566, 13.1047, 16.5856, 9.71009, 21.2508, -9.91968, -3.05793, -28.3953]
[-535.152, 74.5324, 16.7196, -9.10513, 2.03194, -10.2716, 5.86422, 17.6132, -1.67447, 25.2205, -11.9582, -9.62495, -36.1795]
[-534.832, 94.58, 16.7678, -10.1839, 11.5722, -8.37768, 7.56998, 18.684, 6.84446, 19.4338, -20.3944, 5.85976, -21.6592]
[-541.939, 105.854, 2.582, -13.2152, -0.998751, -9.65223, 1.66816, 11.7045, 0.958202, 10.1795, -13.4735, 0.323076, -21.3069]
[-557.338, 116.567, 18.9325, -12.2477, 11.4136, -6.42151, 3.91141, 13.3906, 1.85748, 18.0418, -2.82963, 23.6093, -22.137]
[-557.333, 113.113, 19.7476, -13.0699, 8.88396, -7.57426, 5.43828, 13.5564, -3.24444, 14.6018, -6.68455, 20.1583, -20.2554]
[-521.533, 78.1917, 20.2754, -27.3602, -9.05087, -6.91191, 10.1889, 19.327, 5.51082, -2.77759, -10.4874, 17.3951, -12.8545]
[-487.135, 64.4054, 39.372, -22.0609, -10.153, -14.4954, 8.87789, 42.3528, 26.2456, -10.6034, -14.2779, 22.4984, -1.39246]
[-503.188, 88.5431, 36.5976, -24.0192, -9.48423, -17.0981, -1.74021, 31.6087, 9.88646, -13.8563, -27.4757, 19.1653, -13.4788]
[-526.152, 100.228, 30.0183, -19.6187, -7.01612, -22.5195, -2.41967, 36.9161, 19.1382, -9.25818, -32.8568, 24.158, -5.19042]
[-546.772, 114.271, 38.4308, -27.4553, 9.18841, -18.6456, 11.8459, 38.9588, 12.6151, -4.65447, -34.1896, 26.9072, -21.5543]
[-603.279, 125.792, 68.9516, -60.0422, 11.7244, -14.9915, -7.9859, 17.2039, -18.0642, 6.60621, -23.7851, 16.4236, -11.3687]
[-625.568, 133.096, 70.6208, -28.4733, 22.2955, 1.68351, -8.08299, 5.11235, -9.80408, 5.05816, -12.2634, 16.3428, -5.70496]
mfccframes.txt



In [ ]: