It would be nice to have some documentation on how data is stored, processed, and passed around between the three software packages. I see four situations where data is passed into or out of files.
.aps2 files by QGLlibaps2 and set to the APS2Auspex into .auspex data filesAuspex/Qlab.jl for analysisThese cover ~99% of the situations I can imagine in our normal data flow. The rest of this document is an exploration and note-to-future-self about how this process works and how it could be changed in the future.
The first situation will require QGL to create sequence files for the APS2. Here, we'll create these files and read them back into the workspace to see how they are packed.
In [1]:
import QGL.config
from QGL import *
In [2]:
# a minimal example of a qubit control chain
cl = ChannelLibrary(db_resource_name=":memory:")
q2 = cl.new_qubit("q2")
# specify the particulars for a rack of APS2s
ip_addresses = [f"192.168.1.{i}" for i in [23, 24, 25, 28]]
aps2 = cl.new_APS2_rack("Maxwell", ip_addresses, tdm_ip="192.168.1.11")
aps2.px("TDM").trigger_interval = 500e-6
cl.set_master(aps2.px("TDM"))
# initialize all four APS2 to linear regime
for i in range(1,4):
aps2.tx(i).ch(1).I_channel_amp_factor = 0.5
aps2.tx(i).ch(1).Q_channel_amp_factor = 0.5
aps2.tx(i).ch(1).amp_factor = 1
# create a digitizer - X6 in this case
dig_1 = cl.new_X6("MyX6", address=0)
dig_1.record_length = 1024 + 256
AM2 = cl.new_source("AutodyneM2", "HolzworthHS9000", "HS9004A-492-1",
power=16.0, frequency= 6.74621e9, reference="10MHz")
q2src = cl.new_source("q2source", "HolzworthHS9000", "HS9004A-492-2",
power=16.0, frequency=5.0122e9, reference="10MHz")
cl.set_measure(q2, aps2.tx(2), dig_1.channels[1], gate=False, trig_channel=aps2.tx(2).ch("m2"), generator=AM2)
cl.set_control(q2, aps2.tx(4), generator=q2src)
In [3]:
seqs = [[X(q2),Y(q2),X(q2),MEAS(q2)]]
In [4]:
mf = compile_to_hardware(seqs, 'utils/file_doc')
So now we have .aps2 files in the utils/file_doc folder, along with a .json metafile. mf is a file path to the .json metafile that holds metadata about the sequence and where the .aps2 files are stored. Note, the file extension is set byt the dirver used in QGL. If you were using an APS1 or a APS3, you would see .aps1 or .aps3 files. Not these files are structurally the same, just name differently for clarity.
In [5]:
mf
Out[5]:
In [6]:
import json
In [7]:
md = json.load(open(mf, "r"))
md
Out[7]:
In [8]:
APS2_control_file = md["instruments"]["Maxwell_U4"]
APS2_measure_file = md["instruments"]["Maxwell_U2"]
The metafile is just there for house-keeping across multiple .aps2 files. It also provides a record of what QGL created for each experiment. The point of this notebook is to dive into the .aps2 file structure itself. We can start by looking at the function that creates the file: QGL/QGL/drivers/APS2Pattern.py and the write_sequence_file(). The relevant code is only ~20 lines long:
with open(fileName, 'wb') as FID:
FID.write(b'APS2') # target hardware
FID.write(np.float32(4.0).tobytes()) # Version
FID.write(np.float32(4.0).tobytes()) # minimum firmware version
FID.write(np.uint16(2).tobytes()) # number of channels
# FID.write(np.uint16([1, 2]).tobytes()) # channelDataFor
FID.write(np.uint64(instructions.size).tobytes()) # instructions length
FID.write(instructions.tobytes()) # instructions in uint64 form
#Create the groups and datasets
for chanct in range(2):
#Write the waveformLib to file
if wfInfo[chanct][0].size == 0:
#If there are no waveforms, ensure that there is some element
#so that the waveform group gets written to file.
#TODO: Fix this in libaps2
data = np.array([0], dtype=np.int16)
else:
data = wfInfo[chanct][0]
FID.write(np.uint64(data.size).tobytes()) # waveform data length for channel
FID.write(data.tobytes())
Here we can see the data is just byte encoded and written to file in a particular order. First some basic information about the hardware and minimum firmware.
In [9]:
print(b'APS2')
In [10]:
np.float32(4.0).tobytes()
Out[10]:
In [11]:
np.uint16(2).tobytes()
Out[11]:
In [12]:
with open(APS2_control_file, "rb") as f:
print(f.read())
Of course, this is just the most basic look at what's in the file. To pull out more useful information, QGL has some other function for exploring the data inside.
In [13]:
QGL.drivers.APS2Pattern.read_sequence_file(APS2_control_file)
Out[13]:
In [14]:
QGL.drivers.APS2Pattern.read_sequence_file(APS2_measure_file)
Out[14]:
It might be instructive to see what the pulses should look like so we can confirm the data in the files is what was intended.
In [15]:
plot_pulse_files(mf)
Note here, the data returned in the read_sequence_file methods and in plotting is just the waveform data and not the sequence instruction data. QGL has a variety of functions you can use to inspect the sequence data:
In [16]:
QGL.drivers.APS2Pattern.raw_instructions(APS2_control_file)
Out[16]:
In [17]:
QGL.drivers.APS2Pattern.read_instructions(APS2_control_file)
Out[17]:
Users will likely never need to inspect .aps2 files in this detail but the above is illustrative of what information is in the files. See the documentations for more details on what the instructions actually do when decoded by the hardware. The above is a representation of our simple X,Y,X sequence created above.
The data read by libaps2 can be specified by the user manually or read from files like .aps2 above with the load_sequence_file() function inside libaps2:
// Read the header information
char junk[100];
uint64_t buff_length;
uint16_t num_chans;
std::fstream file(seqFile, std::ios::binary | std::ios::in);
if( !file ) throw APS2_SEQFILE_FAIL;
file.read(junk, 12); // Don't need this info
file.read(reinterpret_cast<char *> (&num_chans), sizeof(uint16_t));
// Start anew
clear_channel_data();
// Read the instructions
file.read(reinterpret_cast<char *> (&buff_length), sizeof(uint64_t));
vector<uint64_t> instructions;
instructions.resize(buff_length);
file.read(reinterpret_cast<char *> (instructions.data()), buff_length*sizeof(uint64_t));
write_sequence(instructions);
// Read the waveforms
vector<int16_t> waveform;
for (int chanct = 0; chanct < num_chans; chanct++) {
file.read(reinterpret_cast<char *> (&buff_length), sizeof(uint64_t));
waveform.resize(buff_length);
file.read(reinterpret_cast<char *> (waveform.data()), buff_length*sizeof(int16_t));
set_waveform(chanct, waveform);
}
To manually set the values a user could use the write_sequence function to write sequence instruction information and the set_waveform function to write the waveform data.
The other end of the data taking process involves saving experimental data to file. This data could have a very rich structure and will depend on how the data pipeline was constructed in Auspex and what was saved to file using the pipeline Writer. The writer uses a data structure called AuspexDataContainer specified in auspex/data_format. It creates a directory structure with a .auspex extension where the data files are stored. This choice to use directories was intentionally made to allow for multiple file writers to work at once. The files inside the folder have the following structure:
folder/data_set_name/
data.dat
data_meta.jsonWhere the .dat contains the actual data and the JSON file holds meta data about data in the .dat file. The data files are numpy memmaps as explained in more detail in the next section.
The last situation is where data needs to be read from file after the experiment is over. This could be necessary for plotting, analysis, etc... Auspex has two main functions for doing this: open_data and load_data.
In [18]:
from auspex.analysis.helpers import open_data, load_data
In [19]:
auspex_data = load_data('/Users/mware/test_data.auspex/')
In [20]:
auspex_data
Out[20]:
In [21]:
dat = open_data('/Users/mware/test_data.auspex/', groupname="q2-main", datasetname="data")
# because no file numer is given, this functions the same as load data
In [22]:
dat
Out[22]:
In [23]:
dat, desc = open_data(25, '/Users/mware/IpythonNotebooks/explorations/Tomography/', groupname="q2-main", datasetname="data", date="200428")
In [24]:
dat
Out[24]:
In [25]:
desc
Out[25]:
The .dat files themselves are binary packed memmaps. They can be accessed amnually with numpy.memmap.
In [26]:
test_file = '/Users/mware/test_data.auspex/q2-main/data.dat'
In [27]:
test_data = np.memmap(test_file, dtype=complex, mode='r')
In [28]:
test_data
Out[28]:
In [29]:
np.size(test_data)
Out[29]:
Anyone using Auspex should never have to work at this level. This breakdown of the file structures is presented only for reference.