Assigning Priors

"There are only two problems in inference: how to assign probability distributions, and how to do integrals." - John Skilling

The topic of how to assign priors will re-appear throughout this course, including in this week's homework.

One way to assign priors is to plead ignorance. Not knowing the value of a parameter up to an additive constant indicates that a uniform (i.e. top hat) prior might be appropriate.

Not knowing the value of a parameter up to an multiplicative constant indicates that a prior uniform in the log of the parameter might be appropriate. Equating ${\rm Pr}(x)\,dx = {\rm Pr}(\log{x})\,d \log{x}$ and assigning the above uniform PDF leads to ${\rm Pr}(x) \propto 1/x$, which is sometimes known loosely as the "Jeffreys Prior"

One problem with uninformative priors is that they can lead to parameter space volumes that are unmanageably large.

Q: Consider a uniform joint prior PDF in N parameter dimensions. What fraction of the a priori allowed volume is in a hypercubic shell that has thickness f of the side length?



In [4]:

    
f = 0.01
N = 2
# Compute difference between two hypervolumes:

print("Volume fraction for f =",f," is")









    



('Volume fraction for f =', 0.01, ' is')

This effect can cause real computational problems when attempting to characterize posterior PDFs - you've seen it already, just in our attempts with two-dimensional grids!

Informative Priors

The prior PDF is an opportunity - to include more information into our analysis.

One good prior PDF is the posterior PDF from a previous analysis.

${\rm Pr}(x|d,H) \propto {\rm Pr}(d|x,H)\;{\rm Pr}(x|c,H)$

Note that ${\rm Pr}(x|c,H)\propto {\rm Pr}(c|x,H)\;{\rm Pr}(x|H)$ by Bayes Theorem, so that using posteriors as priors is equivalent to joint inference from multiple datasets:

${\rm Pr}(x|d,H) \propto {\rm Pr}(d|x,H)\;{\rm Pr}(c|x,H)\;{\rm Pr}(x|H)$

Another good prior PDF is one that accurately represents your beliefs. Such "subjective priors" are typically treated with justified caution, because scientists aspire to perform objective analyses. However, a prior PDF assignment is just one type of assumption in the many that are involved in the model, and they can and should be tested in the same way. We'll return to this in week 3.



In [ ]: