title: "ECE521 Midterm Examination" author: "University of Toronto Faculty of Applied Science and Engineering: Department of Electrical and Computer Engineering" date: "16 February 2017"

$$ $$

Name: __

$$ $$

Student Number: _ Section: $~$ L0101 $~$ or $~$ L0102 $~$ (circle one)

$$ $$$$ $$

Instructions:

• Time allowed: 90 minutes
• Answer all questions. Page xx has space for overflow
• Any questions completed in pencil rather than pen may not be eligible to be remarked even if there was a marking error
• Aids allowed: You are allowed to bring in one $8.5'' \times 11''$ aid sheet double-sided, and a non-programmable calculator

$$ $$

	Question	Value	Mark
	1	xx
	2	xx
	3	xx
	4	xx
	5	xx
	6	xx
	Total	xx

\begin{center}This test should have xx pages including this page \end{center}

Question 1: True or false

Circle one answer (2 marks each, 20 marks total):

---

True False When learning a logistic regression model using SGD, generally it's helpful to centre (zero-mean) the input features True False When using a kNN model based on squared $\ell_2$ distance, generally it's

           helpful to centre the input features

---

\vspace{10cm}

Question 2: Classification

[a] (3 marks) When faced with a multi-class classification problem, describe a disadvantage of using multiple binary classifiers instead of a single multi-class classifier.

(Jimmy: I didn't talk about balance, voting issues, etc in class. Did you? Again maybe we can mention this in the second half and then keep this question for the exam?)

\vspace{2 cm}

[b] (4 marks) In binary classification, suppose $t_m \in \{-1, 1\}$ instead of the usual $\{0, 1\}$. The cross-entropy loss function here is:

$$ \mathcal{L} = - \frac{1}{2}\sum_{m=1}^M \left\{ \log \left[ \text{sigm}(\mathbf{w}^T\mathbf{x}_m)\right] (t_m+1) + \log \left[ 1 - \text{sigm}(\mathbf{w}^T\mathbf{x}_m) \right] (t_m-1) \right\}$$

where $M$ is the number of observations of input-target pairs $(\mathbf{x}_m,t_m)$, and $\text{sigm}(\cdot)$ is the sigmoid function. Note that the first term is zero when $t_m=-1$ and the second term is zero when $t_m=1$. Show that we can rewrite the above loss function as:

$$ \mathcal{L} = \sum_{m=1}^M \log \left[ 1 + \exp(-t_m \mathbf{w}^T\mathbf{x}_m) \right].$$

\vspace{4 cm}

[c] (2 marks) Consider the two choices for $\mathcal{L}$ stated in part [b], Besides the obvious simplicity of the latter expression for $\mathcal{L}$, what other computational advantage does it have over the first $\mathcal{L}$ expression?

\vspace{2 cm}

[d] (3 marks) Explain the reject option of a classifier. Why is it useful?

\vspace{4 cm}

Question 3: Neural networks

[a] (2 marks) When training a large neural network, we commonly determine whether learning has converged by checking (circle all that apply):

\vspace{-.4cm}

\hspace{1cm} [i] $(y_{nk} - t_{nk})^2 \rightarrow 0$ \hspace{1cm} [ii] $\partial E_n(\mathbf{w}) / \partial w_{ji} \rightarrow 0$ \hspace{1cm} [iii] $\text{tanh}(a_j) \rightarrow 0$ \hspace{1cm} [iv] $\delta_j \rightarrow 0$

Here, $y_{nk}$ are model outputs, $t_{nk}$ are targets, $E_n$ is the error for the $n$th data point, $a_j$ is an activation function, and $\delta_j = \partial E_n / \partial a_j$.

(Jimmy: when did you cover this? I don't see slides or an assignment covering it for example. Maybe we could mention it in writing in the second half of the course and then save this question till the exam?) \vspace{1cm}

[b] (3 marks) When designing neural networks, a problem with the sigmoid function's derivative is that it is very small when its input $a \gg 0$ or $a \ll 0$. Suppose we decide to reparameterize a neural network using tanh$(a)$ instead of sigm$(a)$ as the activation functions. Calculate $dh/da$ for $$ h(a) = \text{tanh}(a) = \frac{e^a - e^{-a}}{e^a+e^{-a}}$$ and comment on whether it exhibits a similar problem to the sigmoid function. \vspace{4cm}

[c] (4 marks) Show that there exists a tanh$(a)$ activation function that outputs the same results as the sigmoid function. \vspace{4cm}

[d] Consider the binary threshold neuron, $y = \text{sgn}(\mathbf{w}^T\mathbf{x})$, with no bias $b$ or $w_0$. Consider the set of points $$\{x\} = \{(1, 0, 0), (0, 1, 0), (0, 0, 1), (1, 1, 1) \}$$

[i] Find a three-dimensional parameter vector $\mathbf{w}$ such that the neuron memorizes {t} = {1, 1, 1, 1}. \vspace{1cm}

[ii] Find a three-dimensional parameter vector $\mathbf{w}$ such that the neuron memorizes {t} = {1, 1, 0, 0}. \vspace{1cm}

[iii] Find an unrealizable labelling $\{t\}$. \vspace{1cm}