PHYS366: Statistical Methods in Astrophysics

Lesson 6: Inference in Practice: Evaluating Models

Goals for this session:

• Design and carry out a posterior predictive model check, and describe its relationship to a classical significance test

• Recognize the Bayesian Evidence, know how it is computed, and describe the difference between a Bayes Factor and a likelihood ratio

• Assign priors in various different ways

Related reading:

• Mackay, Chapters 3 and 28

• Gelman et al, Chapters 6 and 7

Model Checking, Expansion and Comparison

• Let's revisit the straight line problem, generating some data and sampling the gradient and intercept again.

• Inferring model parameters involves making assumptions. Let's test those assumptions by checking the model for the accuracy of its predictions.

• If our model is inadequate, we'll need an alternative. How should we compare it to our first one? Accuracy is not the only consideration: efficiency is also important.

• Returning to the straight line problem, let's compare two models via the Bayesian evidence.

• The Bayesian Evidence can be used to good effect in model comparison. It depends on the priors we assume - so let's go back and look at prior assignment in more detail.

• How is model comparison with the Bayesian evidence related to classical hypothesis testing? Let's take a quick look at an example: feature detection.

Recap

• What did we learn?