Numpy arrays are just C arrays wrapped with metadata in Python. Thus, we can share data between C and Python without even copying. In general, this communication is
But without much overhead, we can wrap a shared array (or collection of arrays) in two APIs— one in C++, one in Python— to provide these protections.
commonblock is a nascent library to do this. It passes array lengths and types from Python to C++ via ctypes and uses librt.so (wrapped by prwlock in Python) to implement locks that are usable on both sides.
In [1]:
import numpy
import commonblock
tracks = commonblock.NumpyCommonBlock(
trackermu_qoverp = numpy.zeros(1000, dtype=numpy.double),
trackermu_qoverp_err = numpy.zeros(1000, dtype=numpy.double),
trackermu_phi = numpy.zeros(1000, dtype=numpy.double),
trackermu_eta = numpy.zeros(1000, dtype=numpy.double),
trackermu_dxy = numpy.zeros(1000, dtype=numpy.double),
trackermu_dz = numpy.zeros(1000, dtype=numpy.double),
globalmu_qoverp = numpy.zeros(1000, dtype=numpy.double),
globalmu_qoverp_err = numpy.zeros(1000, dtype=numpy.double))
hits = commonblock.NumpyCommonBlock(
detid = numpy.zeros(5000, dtype=numpy.uint64),
localx = numpy.zeros(5000, dtype=numpy.double),
localy = numpy.zeros(5000, dtype=numpy.double),
localx_err = numpy.zeros(5000, dtype=numpy.double),
localy_err = numpy.zeros(5000, dtype=numpy.double))
CMSSW can be executed within a Python process, thanks to Chris's PR #17236. Since the configuration language is also in Python, you can build the configuration and start CMSSW in the same Python process.
We can get our common block into CMSSW by passing its pointer as part of a ParameterSet. Since this is all one process, that pointer address is still valid when CMSSW launches.
In [2]:
import FWCore.ParameterSet.Config as cms
process = cms.Process("Demo")
process.load("FWCore.MessageService.MessageLogger_cfi")
process.maxEvents = cms.untracked.PSet(input = cms.untracked.int32(1000))
process.source = cms.Source(
"PoolSource", fileNames = cms.untracked.vstring("file:MuAlZMuMu-2016H-002590494DA0.root"))
process.demo = cms.EDAnalyzer(
"DemoAnalyzer",
tracks = cms.uint64(tracks.pointer()), # pass the arrays to C++ as a pointer
hits = cms.uint64(hits.pointer()))
process.p = cms.Path(process.demo)
NumpyCommonBlock.h is a header-only library that defines the interface. We pick up the object by casting the pointer:
tracksBlock = (NumpyCommonBlock*)iConfig.getParameter<unsigned long long>("tracks");
hitsBlock = (NumpyCommonBlock*)iConfig.getParameter<unsigned long long>("hits");and then get safe accessors to each array with a templated method that checks C++'s compiled type against Python's runtime type.
trackermu_qoverp = tracksBlock->newAccessor<double>("trackermu_qoverp");
trackermu_qoverp_err = tracksBlock->newAccessor<double>("trackermu_qoverp_err");
trackermu_phi = tracksBlock->newAccessor<double>("trackermu_phi");
trackermu_eta = tracksBlock->newAccessor<double>("trackermu_eta");
trackermu_dxy = tracksBlock->newAccessor<double>("trackermu_dxy");
trackermu_dz = tracksBlock->newAccessor<double>("trackermu_dz");
globalmu_qoverp = tracksBlock->newAccessor<double>("globalmu_qoverp");
globalmu_qoverp_err = tracksBlock->newAccessor<double>("globalmu_qoverp_err");
detid = hitsBlock->newAccessor<uint64_t>("detid");
localx = hitsBlock->newAccessor<double>("localx");
localy = hitsBlock->newAccessor<double>("localy");
localx_err = hitsBlock->newAccessor<double>("localx_err");
localy_err = hitsBlock->newAccessor<double>("localy_err");Chris's PythonEventProcessor.run() method blocks, so I put it in a thread to let CMSSW and Python run at the same time.
I had to release the GIL with PR #18683 to make this work, and that feature will work its way into releases eventually.
In [3]:
import threading
import libFWCorePythonFramework
import libFWCorePythonParameterSet
class CMSSWThread(threading.Thread):
def __init__(self, process):
super(CMSSWThread, self).__init__()
self.process = process
def run(self):
processDesc = libFWCorePythonParameterSet.ProcessDesc()
self.process.fillProcessDesc(processDesc.pset())
cppProcessor = libFWCorePythonFramework.PythonEventProcessor(processDesc)
cppProcessor.run()
In this demo, I loop over AlCaZMuMu muons and fill the arrays with track parameters (before and after adding muon hits to the fit) and display them as Pandas DataFrames as soon as they're full (before CMSSW finishes).
The idea is that one would stream data from CMSSW into some Python thing in large blocks (1000 tracks/5000 hits at a time in this example).
Bi-directional communication is possible, but I don't know what it could be used for.
In [4]:
cmsswThread = CMSSWThread(process)
cmsswThread.start()
tracks.wait(1) # CMSSW notifies that it has filled the tracks array
tracks.pandas()
Out[4]:
In [5]:
hits.pandas()
Out[5]:
In [7]:
%matplotlib inline
tracks.pandas().plot.hist()
Out[7]:
In [21]:
df = hits.pandas()
df[numpy.abs(df.localy) > 0].plot.hexbin(x="localx", y="localy", gridsize=25)
Out[21]:
This example uses a notify/wait mechanism like basic Java. (The EDAnalyzer says notify(1) when the arrays are full and the Python code above blocks with wait(1).)
This is lock-level programming and is easy to get wrong. Some common use-patterns, like streaming data out of CMSSW in the above example, would be better served by canned classes that look like a Python generator that yields a block of data while CMSSW fills the next. Also, I need to modernize my C++.
In [ ]: