This notebook introduces the new features of PYNQ 2.0 for interacting with the video pipeline. The API has been completely redesigned with high performance image processing applications in mind.
To start, download the base overlay and instantiate the HDMI input and output.
In [1]:
from pynq.overlays.base import BaseOverlay
from pynq.lib.video import *
base = BaseOverlay("base.bit")
hdmi_in = base.video.hdmi_in
hdmi_out = base.video.hdmi_out
In [2]:
hdmi_in.configure()
hdmi_out.configure(hdmi_in.mode)
hdmi_in.start()
hdmi_out.start()
Out[2]:
The monitor should turn on and show a blank screen. To pass the image data through we can tie the output to the input. The tie will last until we send something else to be displayed.
In [3]:
hdmi_in.tie(hdmi_out)
While this provides for a fast way of passing video data through the pipeline there is no way to access or modify the frames. For that we a loop calling readframe and writeframe.
In [4]:
import time
numframes = 600
start = time.time()
for _ in range(numframes):
f = hdmi_in.readframe()
hdmi_out.writeframe(f)
end = time.time()
print("Frames per second: " + str(numframes / (end - start)))
Next we can start adding some OpenCV processing into the mix. For all of these examples we are going to use a Laplacian gradient filter. The first loop is going to perform the grayscale conversion in software.
In [5]:
import cv2
import numpy as np
numframes = 10
grayscale = np.ndarray(shape=(hdmi_in.mode.height, hdmi_in.mode.width),
dtype=np.uint8)
result = np.ndarray(shape=(hdmi_in.mode.height, hdmi_in.mode.width),
dtype=np.uint8)
start = time.time()
for _ in range(numframes):
inframe = hdmi_in.readframe()
cv2.cvtColor(inframe,cv2.COLOR_BGR2GRAY,dst=grayscale)
inframe.freebuffer()
cv2.Laplacian(grayscale, cv2.CV_8U, dst=result)
outframe = hdmi_out.newframe()
cv2.cvtColor(result, cv2.COLOR_GRAY2BGR,dst=outframe)
hdmi_out.writeframe(outframe)
end = time.time()
print("Frames per second: " + str(numframes / (end - start)))
In [6]:
hdmi_out.close()
hdmi_in.close()
By default frames used by the HDMI subsystem are marked as cacheable meaning that the CPU cache is available for speeding up software operation at the expense of needing to flush frames prior to handing them off to the video system. This flushing is handled by PYNQ but can still impose a significant perfomance penalty particularly on 32-bit ARM architectures. To mitigate this the cacheable_frames property can be set to False on the hdmi_in and hdmi_out subsystems. This will improve the performance of passing frames between accelerators at the expense of software libraries operating more slowly or in some cases not working at all.
In [7]:
base.download()
hdmi_in.configure()
hdmi_out.configure(hdmi_in.mode)
hdmi_out.cacheable_frames = False
hdmi_in.cacheable_frames = False
hdmi_out.start()
hdmi_in.start()
Out[7]:
Re-running the plain read-write loop now shows 60 FPS
In [8]:
numframes = 600
start = time.time()
for _ in range(numframes):
f = hdmi_in.readframe()
hdmi_out.writeframe(f)
end = time.time()
print("Frames per second: " + str(numframes / (end - start)))
At the expense of much slower OpenCV performance
In [9]:
numframes = 10
start = time.time()
for _ in range(numframes):
inframe = hdmi_in.readframe()
cv2.cvtColor(inframe,cv2.COLOR_BGR2GRAY,dst=grayscale)
inframe.freebuffer()
cv2.Laplacian(grayscale, cv2.CV_8U, dst=result)
outframe = hdmi_out.newframe()
cv2.cvtColor(result, cv2.COLOR_GRAY2BGR,dst=outframe)
hdmi_out.writeframe(outframe)
end = time.time()
print("Frames per second: " + str(numframes / (end - start)))
In [10]:
hdmi_out.close()
hdmi_in.close()
In [11]:
base.download()
hdmi_in.configure(PIXEL_GRAY)
hdmi_out.configure(hdmi_in.mode)
hdmi_in.cacheable_frames = True
hdmi_out.cacheable_frames = True
hdmi_in.start()
hdmi_out.start()
hdmi_in.tie(hdmi_out)
Now we can rewrite the loop without the software colour conversion.
In [12]:
start = time.time()
numframes = 30
for _ in range(numframes):
inframe = hdmi_in.readframe()
outframe = hdmi_out.newframe()
cv2.Laplacian(inframe, cv2.CV_8U, dst=outframe)
inframe.freebuffer()
hdmi_out.writeframe(outframe)
end = time.time()
print("Frames per second: " + str(numframes / (end - start)))
In [13]:
hdmi_out.close()
hdmi_in.close()
In [14]:
base.download()
hdmi_in.configure(PIXEL_RGB)
hdmi_out.configure(hdmi_in.mode, PIXEL_RGB)
hdmi_in.start()
hdmi_out.start()
hdmi_in.tie(hdmi_out)
This is useful for easily creating and displaying frames with Pillow.
In [15]:
import PIL.Image
frame = hdmi_in.readframe()
image = PIL.Image.fromarray(frame)
image
Out[15]:
An alternative mode is YCbCr which is useful for some image processing algorithms or exporting JPEG files. Because we are not changing the number of bits per pixel we can update the colorspace of the input dynamically.
In [16]:
hdmi_in.colorspace = COLOR_IN_YCBCR
It's probably worth updating the output colorspace as well to avoid the psychedelic effects
In [17]:
hdmi_out.colorspace = COLOR_OUT_YCBCR
Now we can use PIL to read in the frame and perform the conversion back for us.
In [18]:
import PIL.Image
frame = hdmi_in.readframe()
image = PIL.Image.fromarray(frame, "YCbCr")
frame.freebuffer()
image.convert("RGB")
Out[18]:
In [19]:
hdmi_out.close()
hdmi_in.close()
This notebook has only provided an overview of base overlay pipeline. One of the reasons for the changes was to make it easier to add hardware accelerated functions by supporting a wider range of pixel formats without software conversion and separating out the HDMI front end from the video DMA. Explore the code in pynq/lib/video.py for more details.