For unconstrained univariate optimization what are the first order Necessary Conditions for Optimality (FOC). What are the second order optimality conditions (SOC)? Give a mathematical defintion. Also in python, plot the univartiate function $$X^3 -12x^2-6$$ defined over $[-6, 6]$
Also plot its corresponding first and second derivative functions. Eyeballing these graphs, identify candidate optimal points and then classify them as local minimums or maximums. Highlight and label these points in your graphs. Justify your responses using the FOC and SOC.
For unconstrained multi-variate optimization what are the first order Necessary Conditions for Optimality (FOC). What are the second order optimality conditions (SOC)? Give a mathematical defintion. What is the Hessian matrix in this context?
Fill in the BLANKS here: Convex minimization, a subfield of optimization, studies the problem of minimizing BLANK functions over BLANK sets. The BLANK property can make optimization in some sense "easier" than the general case - for example, any local minimum must be a global minimum.
The learning objective function for weighted ordinary least squares (WOLS) (aka weight linear regression) is defined as follows:
$$0.5* \sum_{i=1}^n (weight_i * (W * X_i - y_i)^2)$$Where training set consists of input variables X ( in vector form) and a target variable y, and W is the vector of coefficients for the linear regression model.
Derive the gradient for this weighted OLS by hand; showing each step and also explaining each step.
Write a MapReduce job in MRJob to do the training at scale of a weighted OLS model using gradient descent.
Generate one million datapoints just like in the following notebook: http://nbviewer.ipython.org/urls/dl.dropbox.com/s/kritdm3mo1daolj/MrJobLinearRegressionGD.ipynb
Weight each example as follows:
$$weight(x)= abs(1/x)$$Sample 1% of the data in MapReduce and use the sampled dataset to train a (weighted if available in SciKit-Learn) linear regression model locally using SciKit-Learn (http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)
Plot the resulting weighted linear regression model versus the original model that you used to generate the data. Comment on your findings.
Using MRJob and in Python, plot the error surface for the weighted linear regression model using a heatmap and contour plot. Also plot the current model in the original domain space. (Plot them side by side if possible) Plot the path to convergence (during training) for the weighted linear regression model in plot error space and in the original domain space. Make sure to label your plots with iteration numbers, function, model space versus original domain space, etc. Comment on convergence and on the mean squared error using your weighted OLS algorithm on the weighted dataset versus using the weighted OLS algorithm on the uniformly weighted dataset.
Using the following notebook as a starting point:
Improve this notebook as follows: -- Add in equations into the notebook (not images of equations) -- Number the equations -- Make sure the equation notation matches the code and the code and comments refer to the equations numbers -- Comment the code -- Rename/Reorganize the code to make it more readable -- Rerun the examples similar graphics (or possibly better graphics)
Implement the EM clustering algorithm to determine Bernoulli Mixture Model for discrete data in MRJob.
As a unit test use the dataset in the following slides:
Cross-check that you get the same cluster assignments and cluster Bernouilli models as presented in the slides after 25 iterations. Dont forget the smoothing.
As a full test: use the same dataset from HW 4.5, the Tweet Dataset. Using this data, you will implement a 1000-dimensional EM-based Bernoulli Mixture Model algorithm in MrJob on the users by their 1000-dimensional word stripes/vectors using K = 4. Use the same smoothing as in the unit test.
Repeat this experiment using your KMeans MRJob implementation fron HW4. Report the rand index score using the class code as ground truth label for both algorithms and comment on your findings.
Here is some more information on the Tweet Dataset.
Here you will use a different dataset consisting of word-frequency distributions for 1,000 Twitter users. These Twitter users use language in very different ways, and were classified by hand according to the criteria:
0: Human, where only basic human-human communication is observed.
1: Cyborg, where language is primarily borrowed from other sources (e.g., jobs listings, classifieds postings, advertisements, etc...).
2: Robot, where language is formulaically derived from unrelated sources (e.g., weather/seismology, police/fire event logs, etc...).
3: Spammer, where language is replicated to high multiplicity (e.g., celebrity obsessions, personal promotion, etc... )
Check out the preprints of recent research, which spawned this dataset:
http://arxiv.org/abs/1505.04342 http://arxiv.org/abs/1508.01843
The main data lie in the accompanying file:
topUsers_Apr-Jul_2014_1000-words.txt
and are of the form:
USERID,CODE,TOTAL,WORD1_COUNT,WORD2_COUNT,... . .
where
USERID = unique user identifier CODE = 0/1/2/3 class code TOTAL = sum of the word counts
Using this data, you will implement a 1000-dimensional K-means algorithm in MrJob on the users by their 1000-dimensional word stripes/vectors using several centroid initializations and values of K.
Predict the year of the song. Ask Jimi