This notebook goes into detail about the stages of the video pipeline in the base overlay and is written for people who want to create and integrate their own video IP. For most regular input and output use cases the high level wrappers of HDMIIn and HDMIOut should be used.
Both the input and output pipelines in the base overlay consist of four stages, an HDMI frontend, a colorspace converter, a pixel format converter, and the video DMA. For the input the stages are arranged Frontend -> Colorspace Converter -> Pixel Format -> VDMA with the order reversed for the output side. The aim of this notebook is to give you enough information to use each stage separately and be able to modify the pipeline for your own ends.
Before exploring the pipeline we'll import the entire pynq.lib.video module where all classes relating to the pipelines live. We'll also load the base overlay to serve as an example.
The following table shows the IP responsible for each stage in the base overlay which will be referenced throughout the rest of the notebook
| Stage | Input IP | Output IP |
|---|---|---|
| Frontend (Timing) | video/hdmi_in/frontend/vtc_in |
video/hdmi_out/frontend/vtc_out |
| Frontend (Other) | video/hdmi_in/frontend/axi_gpio_hdmiin |
video/hdmi_out/frontend/axi_dynclk |
| Colour Space | video/hdmi_in/color_convert |
video/hdmi_out/color_convert |
| Pixel Format | video/hdmi_in/pixel_pack |
video/hdmi_outpixel_unpack |
| VDMA | video/axi_vdma |
video/axi_vdam |
In [1]:
from pynq.overlays.base import BaseOverlay
from pynq.lib.video import *
base = BaseOverlay("base.bit")
In [2]:
hdmiin_frontend = base.video.hdmi_in.frontend
Creating the device will signal to the computer that a monitor is connected. Starting the frontend will wait attempt to detect the video mode, blocking until a lock can be achieved. Once the frontend is started the video mode will be available.
In [3]:
hdmiin_frontend.start()
hdmiin_frontend.mode
Out[3]:
The HDMI output frontend can be accessed in a similar way.
In [4]:
hdmiout_frontend = base.video.hdmi_out.frontend
and the mode must be set prior to starting the output. In this case we are just going to use the same mode as the input.
In [5]:
hdmiout_frontend.mode = hdmiin_frontend.mode
hdmiout_frontend.start()
Note that nothing will be displayed on the screen as no video data is currently being send.
The colorspace converter operates on each pixel independently using a 3x4 matrix to transform the pixels. The converter is programmed with a list of twelve coefficients in the folling order:
| in1 | in2 | in3 | 1 | |
|---|---|---|---|---|
| out1 | c1 | c2 | c3 | c10 |
| out2 | c4 | c5 | c6 | c11 |
| out3 | c7 | c8 | c9 | c12 |
Each coefficient should be a floating point number between -2 and +2.
The pixels to and from the HDMI frontends are in BGR order so a list of coefficients to convert from the input format to RGB would be:
[0, 0, 1,
0, 1, 0,
1, 0, 0,
0, 0, 0]
reversing the order of the pixels and not adding any bias.
The driver for the colorspace converters has a single property that contains the list of coefficients.
In [6]:
colorspace_in = base.video.hdmi_in.color_convert
colorspace_out = base.video.hdmi_out.color_convert
bgr2rgb = [0, 0, 1,
0, 1, 0,
1, 0, 0,
0, 0, 0]
colorspace_in.colorspace = bgr2rgb
colorspace_out.colorspace = bgr2rgb
colorspace_in.colorspace
Out[6]:
The pixel format converters convert between the 24-bit signal used by the HDMI frontends and the colorspace converters to either an 8, 24, or 32 bit signal. 24-bit mode passes the input straight through, 32-bit pads the additional pixel with 0 and 8-bit mode selects the first channel in the pixel. This is exposed by a single property to set or get the number of bits.
In [7]:
pixel_in = base.video.hdmi_in.pixel_pack
pixel_out = base.video.hdmi_out.pixel_unpack
pixel_in.bits_per_pixel = 8
pixel_out.bits_per_pixel = 8
pixel_in.bits_per_pixel
Out[7]:
The final element in the pipeline is the video DMA which transfers video frames to and from memory. The VDMA consists of two channels, one for each direction which operate completely independently. To use a channel its mode must be set prior to start being called. After the DMA is started readframe and writeframe transfer frames. Frames are only transferred once with the call blocking if necessary. asyncio coroutines are available as readframe_async and writeframe_async which yield instead of blocking. A frame of the size of the output can be retrieved from the VDMA by calling writechannel.newframe(). This frame is not guaranteed to be initialised to blank so should be completely written before being handed back.
In [8]:
inputmode = hdmiin_frontend.mode
framemode = VideoMode(inputmode.width, inputmode.height, 8)
vdma = base.video.axi_vdma
vdma.readchannel.mode = framemode
vdma.readchannel.start()
vdma.writechannel.mode = framemode
vdma.writechannel.start()
In [9]:
frame = vdma.readchannel.readframe()
vdma.writechannel.writeframe(frame)
In this case, because we are only using 8 bits per pixel, only the red channel is read and displayed.
The two channels can be tied together which will ensure that the input is always mirrored to the output
In [10]:
vdma.readchannel.tie(vdma.writechannel)
The VDMA driver has a strict method of frame ownership. Any frames returned by readframe or newframe are owned by the user and should be destroyed by the user when no longer needed by calling frame.freebuffer(). Frames handed back to the VDMA with writeframe are no longer owned by the user and should not be touched - the data may disappear at any time.
In [11]:
vdma.readchannel.stop()
vdma.writechannel.stop()