The structure of this guide has been introduce the minimal concepts required to easily understandable upper limits. Throughout we provide a constant stream of useful example codes segments. In this way we hope that all the concepts that can sometimes appear quite abstract will be more tractable. We therefore endeavour to have a basic coded example for every technique we introduce throughout the notebooks. If additional examples are required or something is not clear please contact us or raise an issue.
Another philosophy of this collection of notes is to try to explain complex statistical techniques in simple language. We will also introduce technical language but accompany this with a simplified explanation for ease of reading. In addition we want to keep mathematical rigour but it is not always essential to understand these steps. We try to make it clear throughout the text when the maths is supplementary and when its essential to understanding.
Upon first sight an exclusion limit is used to rule out parts of the parameter space, usually part of a favourite model or perhaps the best hope of a scientific communities. Either way, placing limits is an almost exclusively model dependent act. By model here we simply mean a mathematical description of the a theory to explain the data on hand. Now, usually a model has many parameters which presents problems when you want to display an exclusion limit on a 2D plot. For simplicity we will therefore stick to looking at a basic example with just two parameters like the ones considered in most dark matter searches.
Let’s take a look at what a dark matter limit plot might look like. The Fermi-LAT telescope performs observations of our local dwarf galaxies from which we expect a gamma ray signal from dark matter annihilation. Now lets not get into what that means and only use this as a signal (s) vs background (b) discussion. Taking a look at the plot we see two parameters on the x and y axis, the dark matter mass and the velocity averaged cross section respectively. Both of these parameters appear in the calculation required to model the expected signal which is precisely the important step, and often the most complicated. The physical modelling of a signal and background is not something we will cover here and is typically very situation dependent. Modelling for particle physics is covered much more closely here.
The correct way to read a plot like this is to take a value of the dark matter mass (x axis) and trace vertically until you meet the full black line. The point at which you meet the full black line is then your exclusion limit. The paper quotes this line as a 95% confidence level limit meaning that if 100 identical Fermi-LAT experiments were to take a measurement, 95 of the calculated limits would lie in the yellow band and the other 5 would be outside of this band. If you happened to calculate a limit below the yellow band in this plot (meaning you have a stronger signal) then it could either be a statistical downward fluctuation of the background or a hint of some signal component but one must then be careful when claiming how significant the signal is. It is NOT correct to then say that you have a $2\sigma$ detection and it would require a more detailed analysis to determine the exact significance. Another way to phrase this is that at 95% confidence the signal received is consistent with background only.
One important point to notice here is despite the fact the above plot is 2D, the limits are in fact just a series of 1D results stitched together to form a 2D plot. It might seem unimportant but the method for setting exclusion limits which we discuss below will be mostly 1D and strictly speaking it is incorrect to read this as a 2D limit.