CSAL4243: Introduction to Machine Learning

Muhammad Mudassir Khan (mudasssir.khan@ucp.edu.pk)

Lecture 6: Polynomial Regression and Intro to Classification

Overview

• Machine Learning pipeline
• Linear Regression with one variable
  • Model Representation
  • Vectorize Model
• Linear Regression with multiple variables
  • Cost Function
  • Gradient Descent
• Speed up gradient descent
  • Feature Scaling
  • Mean Normalization
  • Combining Both
  • Learning Rate $\alpha$
  • Automatic Covergence Test
• Linear Regression with Multiple Variables Example
  • Read data
  • Feature Scaling and Mean Normalization
  • Initialize Hyper Parameters
  • Model/Hypothesis Function
  • Cost Function
  • Gradient Descent
  • Run Gradient Descent
  • Plot Convergence
  • Predict output using trained model
• Resources
• Credits

Linear Regression with multiple variables

Model $h(x) = \theta_0 + \theta_1 x_1 + \theta_2 x_2 .... + \theta_n x_n$

Psuedocode

Linear Regressio with Gradient Descent

• Load training data into X_train and y_train
• [Optionally] normalize features X_train using $x^i = \frac{x^i - \mu^i}{\rho^i}$ where $\mu^i$ is mean and $\rho^i$ is standard deviation of feature $i$
• Initialize hyperparameters
  • iterations
  • learning rate $\alpha$
• Initialize $\theta_s$
• At each iteration
  • Compute cost using $J(\theta) = \frac{1}{2m}\sum_{i=1}^{m} (h(x^i) - y^i)^2$ where $h(x) = \theta_0 + \theta_1 x_1 + \theta_2 x_2 .... + \theta_n x_n$
  • Update $\theta_s$ using $\begin{align*} \; \; & \theta_j := \theta_j - \alpha \frac{1}{m} \sum\limits_{i=1}^{m} (h_\theta(x_{i}) - y_{i}) \cdot x^j_{i} \; & & \text{for j := 0...n} \end{align*}$
  • [Optionally] Break if cost $J(\theta)$ does not change.

Polynomial Regression

Example

Classification vs Regression

Classification Examples

1. There will be rain tomorrow or not
2. Email is spam or not
3. Bank transaction is fraud or not
4. Image is cat / dog

K - Nearest Neighbor Classifier

Nearest Neighbor Classifier

Resources

Course website: https://w4zir.github.io/ml17s/

Course resources

Credits

Raschka, Sebastian. Python machine learning. Birmingham, UK: Packt Publishing, 2015. Print.

Andrew Ng, Machine Learning, Coursera

Lucas Shen

David Kaleko