The PYNQ-Z1 board contains a HDMI input port, and a HDMI output port connected to the FPGA fabric of the Zynq chip. This means to use the HDMI ports, HDMI controllers must be included in a hardware library or overlay.
The base overlay contains a HDMI input controller, and a HDMI Output controller, both connected to their corresponding HDMI ports. A frame can be captured from the HDMI input, and streamed into DDR memory. The frames in DDR memory, can be accessed from Python.
A framebuffer can be shared between HDMI in and HDMI out to enable streaming.
The overlay contains two video controllers, HDMI in and out. Both interfaces can be controlled independently, or used in combination to capture an image from the HDMI, process it, and display it on the HDMI out.
There is also a USB controller connected to the Zynq PS. A webcam can also be used to capture images, or video input, that can be processed and displayed on the HDMI out.
To use the HDMI in controller, connect the on-board HDMI In port to a valid video source. E.g. your laptop can be used if it has HDMI out. Any HDMI video source can be used up to 1080p.
To use the HDMI in, ensure you have connected a valid HDMI source and execute the next cell. If a valid HDMI source is not detected, the HDMI in controller will timeout with an error.
In [1]:
from pynq import Overlay
from pynq.drivers.video import HDMI
# Download bitstream
Overlay("base.bit").download()
# Initialize HDMI as an input device
hdmi_in = HDMI('in')
The HDMI() argument ‘in’ indicates that the object is in capture mode.
When a valid video input source is connected, the controller should recognize it and start automatically. If a HDMI source is not connected, the code will time-out with an error.
You can manually start/stop the controller
In [2]:
hdmi_in.start()
In [3]:
hdmi_in.stop()
In [4]:
state = hdmi_in.state()
print(state)
The state is returned as an integer value, with one of three possible values:
You can also check the width and height of the input source (assuming a source is connected):
In [5]:
hdmi_in.start()
width = hdmi_in.frame_width()
height = hdmi_in.frame_height()
print('HDMI is capturing a video source of resolution {}x{}'\
.format(width,height))
In [6]:
hdmi_in.frame_index()
Out[6]:
The frame_index() method can also be used to set a new index, if you specify an argument with the method call. For instance:
In [7]:
index = hdmi_in.frame_index()
hdmi_in.frame_index(index + 1)
This will set the current frame index to the next in the sequence. Note that, if index is 2 (the last frame in the list), (index+1) will cause an exception.
If you want to set the next frame in the sequence, use:
In [8]:
hdmi_in.frame_index_next()
Out[8]:
This will loop through the frame list and it will also return the new index as an integer.
In [9]:
from IPython.display import Image
frame = hdmi_in.frame()
orig_img_path = '/home/xilinx/jupyter_notebooks/Getting_Started/images/hdmi_in_frame0.jpg'
frame.save_as_jpeg(orig_img_path)
Image(filename=orig_img_path)
Out[9]:
This will dump the frame as a list _frame[height, width][rgb]. Where rgb is a tuple (r,g,b). If you want to modify the green component of a pixel, you can do it as shown below. In the example, the top left quarter of the image will have the green component increased.
In [10]:
for x in range(int(width/2)):
for y in range(int(height/2)):
(red,green,blue) = frame[x,y]
green = green*2
if(green>255):
green = 255
frame[x,y] = (red, green, blue)
new_img_path = '/home/xilinx/jupyter_notebooks/Getting_Started/images/hdmi_in_frame1.jpg'
frame.save_as_jpeg(new_img_path)
Image(filename=new_img_path)
Out[10]:
This frame() method is a simple way to capture pixel data, but processing it in Python will be slow. If you want to dump a frame at a specific index, just pass the index as an argument of the frame() method:
In [11]:
# dumping frame at index 2
frame = hdmi_in.frame(2)
If higher performance is required, the frame_raw() method can be used:
In [12]:
# dumping frame at current index
frame_raw = hdmi_in.frame_raw()
# dumping frame at index 2
frame_raw = hdmi_in.frame_raw(2)
This method will return a fast memory dump of the internal frame list, as a mono-dimensional list of dimension frame[1920*1080*3] (This array is of fixed size regardless of the input source resolution). 1920x1080 is the maximum supported frame dimension and 3 separate values for each pixel (Blue, Green, Red).
When the resolution is less than 1920x1080, the user must manually extract the correct pixel data.
For example, if the resolution of the video input source is 800x600, meaningful values will only be in the range frame_raw[1920*i*3] to frame_raw[(1920*i + 799)*3] for each i (rows) from 0 to 599. Any other position outside of this range will contain invalid data.
In [13]:
# printing the green component of pixel (0,0)
print(frame_raw[1])
# printing the blue component of pixel (1,399)
print(frame_raw[1920 + 399 + 0])
# printing the red component of the last pixel (599,799)
print(frame_raw[1920*599 + 799 + 2])
In [14]:
from pynq.drivers import HDMI
hdmi_out = HDMI('out')
For the HDMI controller, you have to start/stop the device explicitly:
In [15]:
hdmi_out.start()
In [16]:
hdmi_out.stop()
To check the state of the controller:
In [17]:
state = hdmi_out.state()
print(state)
The state is returned as an integer value, with 2 possible values:
After initialization, the display resolution is set at the lowest level: 640x480 at 60Hz.
To check the current resolution:
In [18]:
print(hdmi_out.mode())
This will print the current mode as a string. To change the mode, insert a valid index as an argument when calling mode():
In [19]:
hdmi_out.mode(4)
Out[19]:
Valid resolutions are:
To use the HDMI input and output to capture and display an image, make both the HDMI input and output share the same frame list. The frame list in both cases can be accessed. You can make the two object share the same frame list by a frame list as an argument to the second object’s constructor.
In [20]:
from pynq.drivers.video import HDMI
hdmi_in = HDMI('in')
hdmi_out = HDMI('out', frame_list=hdmi_in.frame_list)
hdmi_out.mode(4)
Out[20]:
To start the controllers:
In [21]:
hdmi_out.start()
hdmi_in.start()
The last step is always to stop the controllers and delete HDMI objects.
In [22]:
hdmi_out.stop()
hdmi_in.stop()
del hdmi_out
del hdmi_in