Home Assignment No. 3: Part 1

In this part of the homework you are to solve several problems related to machine learning algorithms.

• For every separate problem you can get only 0 points or maximal points for this problem. There are NO INTERMEDIATE scores.
• Your solution must me COMPLETE, i.e. contain all required formulas/proofs/detailed explanations.
• You must write your solution for any problem just right after the words BEGIN SOLUTION. Attaching pictures of your handwriting is allowed, but highly discouraged.
• If you want an easy life, you have to use BUILT-IN METHODS of sklearn library instead of writing tons of our yown code. There exists a class/method for almost everything you can imagine (related to this homework).
• To do some tasks in this part of homework, you have to write CODE directly inside specified places inside notebook CELLS.
• In some problems you may be asked to provide short discussion of the results. In this cases you have to create MARKDOWN cell with your comments right after the your code cell.

• Your SOLUTION notebook MUST BE REPRODUCIBLE, i.e. if the reviewer decides to execute Kernel -> Restart Kernel and Run All Cells, after all the computation he will obtain exactly the same solution (with all the corresponding plots) as in your uploaded notebook. For this purpose, we suggest to fix random seed or (better) define random_state= inside every algorithm that uses some pseudorandomness.

• Your code must be clear to the reviewer. For this purpose, try to include neccessary comments inside the code. But remember: GOOD CODE MUST BE SELF-EXPLANATORY without any additional comments.

• The are problems with * mark - they are not obligatory. You can get EXTRA POINTS for solving them. ## $\LaTeX$ in Jupyter Jupyter has constantly improving $\LaTeX$ support. Below are the basic methods to write neat, tidy, and well typeset equations in your notebooks:
• to write an inline equation use

  $ you latex equation here $
  

• to write an equation, that is displayed on a separate line use

  $$ you latex equation here $$
  

• to write a block of equations use

  \begin{align}
    left-hand-side
        &= right-hand-side on line 1
        \\
        &= right-hand-side on line 2
        \\
        &= right-hand-side on the last line
  \end{align}
  

  The ampersand (&) aligns the equations horizontally and the double backslash (\\) creates a new line.

Write your theoretical derivations within such blocks:

**BEGIN Solution**

<!-- >>> your derivation here <<< -->

**END Solution**

Please, write your implementation within the designated blocks:

...
### BEGIN Solution

# >>> your solution here <<<

### END Solution
...

Bayesian Models. GLM

Task 1 (1 pt.)

Consider a univariate Gaussian distribution $\mathcal{N}(x; \mu, \tau^{-1})$. Let's define Gaussian-Gamma prior for parameters $(\mu, \tau)$:

Find the posterior distribution of $(\mu, \tau)$ after observing $X = (x_1, \dots, x_n)$.

BEGIN Solution

By condition: $$ \mathbb{P}(X | \mu, \tau) = \prod_{i=1}^n \mathbb{P}(x_i | \mu, \tau) $$ As we know distribution of each datasample: $$ \mathbb{P}(X | \mu, \tau) = \prod_{i=1}^n \frac{\tau^{\frac{1}{2}}}{\sqrt{2 \pi}} \exp{ \Big[-\frac{\tau (x_i - \mu)^2 }{2}\Big]} = \frac{\tau^{\frac{n}{2}}}{(2 \pi)^{\frac{n}{2}}} \exp{ \Big[-\frac{\tau}{2} \sum_{i=1}^n(x_i - \mu)^2 \Big]} $$

After simple regrouping: $$ \beta (\mu - \mu_0)^2 + n(\overline{x} - \mu)^2 = (\beta + n) \Big( \mu - \frac{\beta \mu_0 + n \overline{x}}{\beta + n} \Big)^2 + \frac{\beta n ( \overline{x} - \mu_0)^2}{\beta + n} $$

Again regrouping: $$ p(\mu, \tau | X) \propto \tau^{\frac{n}{2} + a - \frac{1}{2}} \exp{\Big[ -\tau \Big[ \frac{1}{2} ns + b + \frac{\beta n ( \overline{x} - \mu_0)^2}{2(\beta + n)} \Big] \Big]} \exp{\Big[ - \frac{\tau}{2} (\beta + n) \Big( \mu - \frac{\beta \mu_0 + n \overline{x}}{\beta + n} \Big)^2 \Big]} $$ Finally: $$ \boxed{ p(\mu, \tau | X) \propto \Gamma\Big(\tau, \frac{n}{2} + a, \frac{1}{2} ns +b + \frac{\beta n ( \overline{x} - \mu_0)^2}{2(\beta + n)} \Big) \mathcal{N}\Big( \mu, \frac{\beta \mu_0 + n \overline{x}}{\beta + n}, (\tau(\beta + n))^{-1}\Big) } $$ END Solution

Task 2 (1 + 1 + 1 = 3 pt.)

Evaluate the following integral using the Laplace approximation: \begin{equation} x \mapsto \int \sigma(w^T x) \mathcal{N}(w; 0, \Sigma) dw \,, \end{equation} for $x = \bigl(\tfrac23, \tfrac16, \tfrac16\bigr)\in \mathbb{R}^3$ and \begin{equation} \Sigma = \begin{pmatrix} 1 & -0.25 & 0.75 \\ -0.25 & 1 & 0.5 \\ 0.75 & 0.5 & 2 \end{pmatrix} \,. \end{equation}

Task 2.1 (1 pt.)

Use the Hessian matrix computed numericaly via finite differences. (Check out Numdifftools)

Task 2.2 (1 pt.)

Use the diagonal approximation of the Hessian computed by autodifferentiation in pytorch.

Task 2.3 (1 pt.)

Compare the results comparing the absolute errors of the results (this is possible with Monte-Carlo estimate of the integral). Write 1-2 sentences in the results discussion.

BEGIN Solution

So, we have got big absolute error in the second line due to the fact that we used Hessian diagonal approximation which neglects a lot of values off the diagonal

END Solution

Gaussian Processes

Task 3 (1 + 2 = 3 pt.)

Task 3.1 (1 pt.)

Assuimng the matrices $A \in \mathbb{R}^{n \times n}$ and $D \in \mathbb{R}^{d \times d}$ are invertible, using gaussian elimination find the inverse matrix for the following block matrix: \begin{equation} \begin{pmatrix} A & B \\ C & D \end{pmatrix} \,, \end{equation} where $C \in \mathbb{R}^{d \times n}$ and $B \in \mathbb{R}^{n \times d}$.

BEGIN Solution

Finally, $$

\boxed {\begin{pmatrix} A & B \\ C & D \end{pmatrix}^{-1}

\Bigg( \begin{array}{cc} A^{-1} + A^{-1}C (D - B A^{-1}C)^{-1} B A^{-1} & - (D - B A^{-1}C)^{-1} BA^{-1} \

• A^{-1} C (D - B A^{-1}C)^{-1} & (D - B A^{-1}C)^{-1}\ \end{array} \Bigg) } $$

END Solution

Task 3.2 (2 pt.)

Assume that the function $y(x)$, $x \in \mathbb{R}^d$, is a realization of the Gaussian Process $GP\bigl(0; K(\cdot, \cdot)\bigr)$ with $K(a, b) = \exp({- \gamma \|a - b\|_2^2}))$.

Suppose two datasets were observed: noiseless ${D_0}$ and noisy ${D_1}$ \begin{aligned} & D_0 = \bigl(x_i, y(x_i) \bigr)_{i=1}^{n} \,, \\ & D_1 = \bigl(x^\prime_i, y(x^\prime_i) + \varepsilon_i \bigr)_{i=1}^{m} \,, \end{aligned} where $\varepsilon_i \sim \text{ iid } \mathcal{N}(0, \sigma^2)$, independent of process $y$.

Derive the conditional distribution of $y(x) \big\vert_{D_0, D_1}$ at a new $x$.

BEGIN Solution

END Solution

Task 4 (1 + 2 = 3 pt.)

Task 4.1 (1 pt.)

In the late 1950’s Charles Keeling invented an accurate way to measure atmospheric $CO_2$ concentration and began taking regular measurements at the Mauna Loa observatory.

Take monthly_co2_mlo.csv file, load it and prepare the data.

1. Load the CO2 [ppm] time series
2. Replace $-99.99$ with NaN and drop the missing observations
3. Split the time series into train and test
4. Normalize the target value by fitting a transformation on the train
5. Plot the resulting target against the time index

Task 4.2 (2 pt.)

Use GPy library for training and prediction. Fit a GP and run the predict on the test. Useful kernels to combine: GPy.kern.RBF, GPy.kern.Poly, GPy.kern.StdPeriodic, GPy.kern.White, GPy.kern.Linear.

1. Plot mean and confidence interval of the prediction.
2. Inspect them on normality by scatter plot: plot predicted points/time series against true values.
3. Estimate the prediction error with r2_score. R2-score accepted > 0.83 on test sample.