(In order to load the stylesheet of this notebook, execute the last code cell in this notebook)

Analyzing hotel ratings on Tripadvisor

In this homework we will focus on practicing two techniques: web scraping and regression. For the first part, we will build upon the sample code from the Lecture and attempt to get some basic information for each hotel. Then, we will fit a regression model on this information and try to analyze it.

One of the main disadvantages of scraping a website instead of using an API is that, without any notice, the website may change its layout and render our code useless. Something like that happened in our case. Tripadvisor changed the layout of the buttons that we use to navigate between the different pages of the results. This was the main reason people were having problem with executing the code.

Task 1 (20 pts)

The first task of the homework is to fix the scraping code. We basically need to replace the part where we are checking if there is another page and getting its link with new code that reflects the new navigation layout.

Task 2 (30 pts)

Then, for each hotel that our search returns, we will "click" (with the code of course) on it and scrape the information below.

Of course, feel free to collect even more data if you want.

Task 3 (20 pts)

Now, we will use regression to analyze this information. First, we will fit a linear regression model that predicts the average rating. For example, for the hotel above, the average rating is

Use the model to analyze the important factors that decide the $\text{AVG_SCORE}$.

Task 4 (30 pts)

Finally, we will use logistic regression to decide if a hotel is excellent or not. We classify a hotel as excellent if more than 60% of its ratings are 5 stars. This is a binary attribute on which we can fit a logistic regression model. As before, use the model to analyze the data.

In order to use code from a Python script file, we need to put that file in the same folder as the notebook and import it as a library. Then, we will be able to access it's functions. For example, in the case of the lecture code, we could do the following:

import scrape_solution as scrape

scrape.get_city_page()

Of course, you might need to modify and restructure the code so that it returns what you need.